Skip to content

Add script to generate pycfmodel resources from CloudFormation schemas#200

Open
Jordi Soucheiron (jsoucheiron) wants to merge 9 commits intomasterfrom
resource-generator-script
Open

Add script to generate pycfmodel resources from CloudFormation schemas#200
Jordi Soucheiron (jsoucheiron) wants to merge 9 commits intomasterfrom
resource-generator-script

Conversation

@jsoucheiron
Copy link
Member

@jsoucheiron Jordi Soucheiron (jsoucheiron) commented Feb 2, 2026

Summary

Adds a CLI script that generates static pycfmodel resource classes from AWS CloudFormation schemas, plus the top 10 most common unmodeled resource types (increasing coverage from 51.9% to 71.8%).

Usage

# Generate a Lambda function resource
python scripts/generate_resource_from_schema.py AWS::Lambda::Function

# Output to the resources directory
python scripts/generate_resource_from_schema.py AWS::Lambda::Function --output-dir pycfmodel/model/resources

# Preview without writing files
python scripts/generate_resource_from_schema.py AWS::Lambda::Function --dry-run

# List all available resource types (1510 types)
python scripts/generate_resource_from_schema.py --list-types

# Generate multiple resources at once
python scripts/generate_resource_from_schema.py AWS::DynamoDB::Table AWS::SQS::Queue -o pycfmodel/model/resources

New Resource Types Added

Based on analysis of 2,211 production CloudFormation templates containing 5,285 resources:

Resource Type Count Cumulative Coverage
AWS::CloudWatch::Alarm 165 55.1%
AWS::IAM::InstanceProfile 146 57.8%
AWS::Route53::RecordSet 134 60.4%
AWS::SQS::Queue 125 62.7%
AWS::SNS::Topic 99 64.6%
AWS::SNS::Subscription 79 66.1%
AWS::AutoScaling::AutoScalingGroup 78 67.6%
AWS::ElasticLoadBalancingV2::Listener 78 69.0%
AWS::ElasticLoadBalancingV2::TargetGroup 75 70.5%
AWS::DynamoDB::Table 72 71.8%

Coverage improvement: 51.9% → 71.8% (1,051 additional resources now properly typed)

Generator Features

  • Downloads CloudFormation schemas from AWS (cached during execution)
  • Generates Properties class with proper type annotations (ResolvableStr, ResolvableGeneric, etc.)
  • Generates Resource class inheriting from Resource base class
  • Follows existing pycfmodel conventions (naming, structure, docstrings)
  • Handles optional vs required properties correctly
  • --dry-run option to preview generated code
  • --list-types option to list all 1510 available AWS resource types

Example Generated Code

class SQSQueueProperties(CustomModel):
    """Properties for AWS::SQS::Queue."""

    ContentBasedDeduplication: Optional[ResolvableBool] = None
    DeduplicationScope: Optional[ResolvableStr] = None
    DelaySeconds: Optional[ResolvableInt] = None
    FifoQueue: Optional[ResolvableBool] = None
    QueueName: Optional[ResolvableStr] = None
    # ... more properties


class SQSQueue(Resource):
    """AWS SQS Queue resource."""

    Type: Literal["AWS::SQS::Queue"]
    Properties: Optional[Resolvable[SQSQueueProperties]] = None

Test plan

  • Generated code compiles without errors
  • Generated classes inherit from Resource correctly
  • Generated classes can be instantiated with CloudFormation properties
  • Generated classes support intrinsic functions (Ref, Fn::Sub, Fn::GetAtt)
  • Generated classes have standard Resource attributes (DependsOn, Condition, etc.)
  • Generated code matches existing pycfmodel naming conventions
  • CLI options work correctly (--list-types, --dry-run, --output-dir)
  • All 665 existing tests pass

🤖 Generated with Claude Code

Copilot AI review requested due to automatic review settings February 2, 2026 19:13
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a CLI script that generates static pycfmodel resource classes from AWS CloudFormation schemas, enabling permanent addition of new AWS resource types to the codebase.

Changes:

  • Adds a script (generate_resource_from_schema.py) that downloads CloudFormation schemas from AWS and generates Python resource classes following pycfmodel conventions
  • Includes comprehensive test coverage for the generator, validating code compilation, functionality, structure, and CLI interface
  • Supports generating multiple resources with options for dry-run preview and listing all 1510+ available AWS resource types

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File Description
scripts/generate_resource_from_schema.py Main script implementing schema download, type conversion, and code generation functionality
tests/test_resource_generator.py Test suite covering compilation, functionality, structure, and CLI interface of the generator

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

short_desc = prop_desc[:80] + "..." if len(prop_desc) > 80 else prop_desc
lines.append(f" - {prop_name}: {short_desc}")
lines.append("")
lines.append(f" More info at [AWS Docs](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-{resource_type.lower().replace('::', '-').replace('aws-', '')}.html)")
Copy link

Copilot AI Feb 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The URL construction logic is duplicated on lines 274 and 300. Extract this into a helper function to avoid code duplication.

Copilot uses AI. Check for mistakes.
@jsoucheiron Jordi Soucheiron (jsoucheiron) marked this pull request as draft February 2, 2026 19:44
@jsoucheiron Jordi Soucheiron (jsoucheiron) marked this pull request as ready for review February 2, 2026 21:15
@w0rmr1d3r Ramon (w0rmr1d3r) marked this pull request as draft February 3, 2026 08:35
… schemas

This script creates permanent Python resource files that can be integrated
into the pycfmodel codebase, as an alternative to dynamic generation at runtime.

Usage:
    python scripts/generate_resource_from_schema.py AWS::Lambda::Function
    python scripts/generate_resource_from_schema.py AWS::Lambda::Function --output-dir pycfmodel/model/resources
    python scripts/generate_resource_from_schema.py --list-types

Features:
- Downloads CloudFormation schemas from AWS (cached during execution)
- Generates Properties class with proper type annotations (ResolvableStr, etc.)
- Generates Resource class inheriting from Resource base class
- Follows existing pycfmodel conventions (naming, structure, docstrings)
- Handles optional vs required properties correctly
- Supports multiple resource types in a single invocation
- --dry-run option to preview generated code
- --list-types option to list all available AWS resource types

The generated code is compatible with existing pycfmodel patterns and can be
used directly in templates, supporting intrinsic functions like Ref, Fn::Sub, etc.

Co-Authored-By: Claude Opus 4.5 <[email protected]>
Generated using the new resource generator script, based on analysis of
2,211 production CloudFormation templates containing 5,285 resources.

New resource types added:
- AWS::CloudWatch::Alarm (165 occurrences)
- AWS::IAM::InstanceProfile (146 occurrences)
- AWS::Route53::RecordSet (134 occurrences)
- AWS::SQS::Queue (125 occurrences)
- AWS::SNS::Topic (99 occurrences)
- AWS::SNS::Subscription (79 occurrences)
- AWS::AutoScaling::AutoScalingGroup (78 occurrences)
- AWS::ElasticLoadBalancingV2::Listener (78 occurrences)
- AWS::ElasticLoadBalancingV2::TargetGroup (75 occurrences)
- AWS::DynamoDB::Table (72 occurrences)

Coverage improvement: 51.9% → 71.8% (1,051 additional resources now typed)

Also includes improvements to the generator script:
- Better handling of multiple JSON types (e.g., ["string", "object"])
- Improved naming conventions for service acronyms (IAM, EC2, RDS, etc.)
- Better filename generation for ELBv2, DynamoDB, etc.

Co-Authored-By: Claude Opus 4.5 <[email protected]>
- Refactor generator to use CodeGenerator class for cleaner code organization
- Add proper handling of $ref definitions for array types (inline them)
- Add proper handling of $ref definitions for primitive types (inline them)
- Use ResolvableGeneric for nested object types due to Pydantic v2 limitations
  with union_mode annotation on model schemas
- Add topological sorting infrastructure for future nested type support
- Fix dependency tracking for array items in definitions
- Update test to be more flexible about complex type handling
- Regenerate all 10 resource models with improved type handling

The script now properly:
- Inlines array definitions as List[ItemType]
- Inlines primitive type definitions as ResolvableStr/Int/Bool
- Uses ResolvableGeneric for complex object references
- Tracks dependencies correctly through array items

Co-Authored-By: Claude Opus 4.5 <[email protected]>
Updated the following resources with fields that were added to the AWS
CloudFormation schemas since the models were originally created:

- IAMUser: Added Tags field
- KMSKey: Added RotationPeriodInDays field
- S3Bucket: Added AbacStatus, MetadataConfiguration, MetadataTableConfiguration
- OpenSearchDomain: Added AIMLOptions, IPAddressType, IdentityCenterOptions,
  OffPeakWindowOptions, SkipShardMigrationWait, SoftwareUpdateOptions
- EC2VPCEndpoint: Added DnsOptions, IpAddressType, ResourceConfigurationArn,
  ServiceNetworkArn, ServiceRegion, Tags

Co-Authored-By: Claude Opus 4.5 <[email protected]>
The standard Resolvable[T] type uses Field(union_mode="left_to_right")
which cannot be applied to Pydantic model schemas defined in the same file.
This prevented generated resource models from having properly typed nested
structures like KeySchema, AvailabilityZoneImpairmentPolicy, etc.

This commit introduces ResolvableModel(T), a new type helper that:
- Uses a custom __get_pydantic_core_schema__ to handle Model | FunctionDict unions
- Bypasses Pydantic's union_mode limitation
- Works with nested models defined in the same file
- Properly handles CloudFormation intrinsic functions (Ref, Fn::*, etc.)

The generator script now:
- Creates nested model classes (e.g., KeySchema, AttributeDefinition)
- Generates ResolvableX type aliases using ResolvableModel()
- Uses these type aliases for nested object references

Benefits:
- Nested models are properly typed (not ResolvableGeneric)
- IDE autocompletion works for nested fields
- Type checking catches errors at development time
- CloudFormation intrinsic functions still work correctly

Example:
  # Before: table.Properties.KeySchema[0] was Generic
  # After:  table.Properties.KeySchema[0] is KeySchema
  #         with AttributeName and KeyType fields accessible

Co-Authored-By: Claude Opus 4.5 <[email protected]>
Applied the new ResolvableModel type to existing models that were
using ResolvableGeneric for nested structures. This provides proper
typing and IDE support for nested CloudFormation resource properties.

Changes:

Auto-generated resources (regenerated with nested types):
- AWS::S3::Bucket - 50+ nested model types (CorsConfiguration,
  ReplicationConfiguration, LifecycleConfiguration, etc.)

Manually written resources (added typed nested models):
- AWS::OpenSearchService::Domain - ClusterConfig, EBSOptions,
  VPCOptions, AdvancedSecurityOptions, etc.
- AWS::Elasticsearch::Domain - Same nested types as OpenSearch
- AWS::EC2::VPCEndpoint - DnsOptions, Tags
- AWS::IAM::User - Tags (using existing Tag type)

Field type improvements:
- TopicARN: ResolvableStr (was ResolvableGeneric)
- KMSMasterKeyId: ResolvableStr (was ResolvableGeneric)
- UserPoolClientId: ResolvableStr (was ResolvableGeneric)
- CertificateArn: ResolvableStr (was ResolvableGeneric)
- Various JSON policies: Resolvable[dict] (was ResolvableGeneric)

Only PolicyDocument fields remain as ResolvableGeneric since they
are intentionally arbitrary JSON (IAM policy documents).

Updated tests to work with the new typed access patterns:
- LogPublishingOptions and AdvancedOptions now use dict access
  instead of attribute access

Co-Authored-By: Claude Opus 4.5 <[email protected]>
1. Generator reuses existing pycfmodel types (Tag):
   - Added REUSE_EXISTING_TYPES mapping in CodeGenerator
   - Tag now imports from pycfmodel.model.resources.properties.tag
   - Avoids generating duplicate Tag classes in each resource

2. Fixed IAM User LoginProfile typing:
   - Added proper LoginProfile model with Password and PasswordResetRequired
   - Updated has_hardcoded_credentials() to work with typed model
   - LoginProfile is now Optional[ResolvableLoginProfile]

3. Fixed ResolvableModel type:
   - Removed None from the Union (Optional is applied at field level)
   - Cleaner type definition: Union[model_cls, FunctionDict]

4. Added comprehensive tests for ResolvableModel:
   - test_resolvable_model_accepts_dict
   - test_resolvable_model_accepts_model_instance
   - test_resolvable_model_accepts_ref/fn_sub/fn_if
   - test_resolvable_model_accepts_none_for_optional
   - test_resolvable_model_rejects_invalid_dict/type
   - test_resolvable_model_serialization (with and without intrinsics)

5. Generator no longer uses ResolvableGeneric:
   - All fallbacks now use Resolvable[dict] instead
   - Cleaner generated code without Generic import

Co-Authored-By: Claude Opus 4.5 <[email protected]>
1. Replaced global variable caching with SchemaRegistry class:
   - Encapsulates schema downloading and caching logic
   - Better testability and cleaner separation of concerns
   - Supports per-region caching

2. Extracted test fixture helper function:
   - Added generate_and_load_module() context manager
   - Eliminates code duplication between test fixtures
   - Proper cleanup with try/finally pattern

3. Removed unused Optional import

Co-Authored-By: Claude Opus 4.5 <[email protected]>
Added tests for 10 resource types that had model implementations
but no corresponding test files:

- test_autoscaling_auto_scaling_group.py
- test_cloudwatch_alarm.py
- test_dynamodb_table.py
- test_elbv2_listener.py
- test_elbv2_target_group.py
- test_iam_instance_profile.py
- test_route53_record_set.py
- test_sns_subscription.py
- test_sns_topic.py
- test_sqs_queue.py

Each test file covers:
- Basic resource instantiation
- Property validation
- Required field validation
- Nested object handling
- Common use cases for the resource type

Co-Authored-By: Claude Opus 4.5 <[email protected]>
@jsoucheiron Jordi Soucheiron (jsoucheiron) marked this pull request as ready for review February 3, 2026 09:46
@jsoucheiron Jordi Soucheiron (jsoucheiron) marked this pull request as draft February 3, 2026 09:49
@jsoucheiron Jordi Soucheiron (jsoucheiron) marked this pull request as ready for review February 4, 2026 14:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant