Add script to generate pycfmodel resources from CloudFormation schemas#200
Open
Jordi Soucheiron (jsoucheiron) wants to merge 9 commits intomasterfrom
Open
Add script to generate pycfmodel resources from CloudFormation schemas#200Jordi Soucheiron (jsoucheiron) wants to merge 9 commits intomasterfrom
Jordi Soucheiron (jsoucheiron) wants to merge 9 commits intomasterfrom
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds a CLI script that generates static pycfmodel resource classes from AWS CloudFormation schemas, enabling permanent addition of new AWS resource types to the codebase.
Changes:
- Adds a script (
generate_resource_from_schema.py) that downloads CloudFormation schemas from AWS and generates Python resource classes following pycfmodel conventions - Includes comprehensive test coverage for the generator, validating code compilation, functionality, structure, and CLI interface
- Supports generating multiple resources with options for dry-run preview and listing all 1510+ available AWS resource types
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| scripts/generate_resource_from_schema.py | Main script implementing schema download, type conversion, and code generation functionality |
| tests/test_resource_generator.py | Test suite covering compilation, functionality, structure, and CLI interface of the generator |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| short_desc = prop_desc[:80] + "..." if len(prop_desc) > 80 else prop_desc | ||
| lines.append(f" - {prop_name}: {short_desc}") | ||
| lines.append("") | ||
| lines.append(f" More info at [AWS Docs](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-{resource_type.lower().replace('::', '-').replace('aws-', '')}.html)") |
There was a problem hiding this comment.
The URL construction logic is duplicated on lines 274 and 300. Extract this into a helper function to avoid code duplication.
… schemas
This script creates permanent Python resource files that can be integrated
into the pycfmodel codebase, as an alternative to dynamic generation at runtime.
Usage:
python scripts/generate_resource_from_schema.py AWS::Lambda::Function
python scripts/generate_resource_from_schema.py AWS::Lambda::Function --output-dir pycfmodel/model/resources
python scripts/generate_resource_from_schema.py --list-types
Features:
- Downloads CloudFormation schemas from AWS (cached during execution)
- Generates Properties class with proper type annotations (ResolvableStr, etc.)
- Generates Resource class inheriting from Resource base class
- Follows existing pycfmodel conventions (naming, structure, docstrings)
- Handles optional vs required properties correctly
- Supports multiple resource types in a single invocation
- --dry-run option to preview generated code
- --list-types option to list all available AWS resource types
The generated code is compatible with existing pycfmodel patterns and can be
used directly in templates, supporting intrinsic functions like Ref, Fn::Sub, etc.
Co-Authored-By: Claude Opus 4.5 <[email protected]>
Generated using the new resource generator script, based on analysis of 2,211 production CloudFormation templates containing 5,285 resources. New resource types added: - AWS::CloudWatch::Alarm (165 occurrences) - AWS::IAM::InstanceProfile (146 occurrences) - AWS::Route53::RecordSet (134 occurrences) - AWS::SQS::Queue (125 occurrences) - AWS::SNS::Topic (99 occurrences) - AWS::SNS::Subscription (79 occurrences) - AWS::AutoScaling::AutoScalingGroup (78 occurrences) - AWS::ElasticLoadBalancingV2::Listener (78 occurrences) - AWS::ElasticLoadBalancingV2::TargetGroup (75 occurrences) - AWS::DynamoDB::Table (72 occurrences) Coverage improvement: 51.9% → 71.8% (1,051 additional resources now typed) Also includes improvements to the generator script: - Better handling of multiple JSON types (e.g., ["string", "object"]) - Improved naming conventions for service acronyms (IAM, EC2, RDS, etc.) - Better filename generation for ELBv2, DynamoDB, etc. Co-Authored-By: Claude Opus 4.5 <[email protected]>
- Refactor generator to use CodeGenerator class for cleaner code organization - Add proper handling of $ref definitions for array types (inline them) - Add proper handling of $ref definitions for primitive types (inline them) - Use ResolvableGeneric for nested object types due to Pydantic v2 limitations with union_mode annotation on model schemas - Add topological sorting infrastructure for future nested type support - Fix dependency tracking for array items in definitions - Update test to be more flexible about complex type handling - Regenerate all 10 resource models with improved type handling The script now properly: - Inlines array definitions as List[ItemType] - Inlines primitive type definitions as ResolvableStr/Int/Bool - Uses ResolvableGeneric for complex object references - Tracks dependencies correctly through array items Co-Authored-By: Claude Opus 4.5 <[email protected]>
Updated the following resources with fields that were added to the AWS CloudFormation schemas since the models were originally created: - IAMUser: Added Tags field - KMSKey: Added RotationPeriodInDays field - S3Bucket: Added AbacStatus, MetadataConfiguration, MetadataTableConfiguration - OpenSearchDomain: Added AIMLOptions, IPAddressType, IdentityCenterOptions, OffPeakWindowOptions, SkipShardMigrationWait, SoftwareUpdateOptions - EC2VPCEndpoint: Added DnsOptions, IpAddressType, ResourceConfigurationArn, ServiceNetworkArn, ServiceRegion, Tags Co-Authored-By: Claude Opus 4.5 <[email protected]>
The standard Resolvable[T] type uses Field(union_mode="left_to_right") which cannot be applied to Pydantic model schemas defined in the same file. This prevented generated resource models from having properly typed nested structures like KeySchema, AvailabilityZoneImpairmentPolicy, etc. This commit introduces ResolvableModel(T), a new type helper that: - Uses a custom __get_pydantic_core_schema__ to handle Model | FunctionDict unions - Bypasses Pydantic's union_mode limitation - Works with nested models defined in the same file - Properly handles CloudFormation intrinsic functions (Ref, Fn::*, etc.) The generator script now: - Creates nested model classes (e.g., KeySchema, AttributeDefinition) - Generates ResolvableX type aliases using ResolvableModel() - Uses these type aliases for nested object references Benefits: - Nested models are properly typed (not ResolvableGeneric) - IDE autocompletion works for nested fields - Type checking catches errors at development time - CloudFormation intrinsic functions still work correctly Example: # Before: table.Properties.KeySchema[0] was Generic # After: table.Properties.KeySchema[0] is KeySchema # with AttributeName and KeyType fields accessible Co-Authored-By: Claude Opus 4.5 <[email protected]>
Applied the new ResolvableModel type to existing models that were using ResolvableGeneric for nested structures. This provides proper typing and IDE support for nested CloudFormation resource properties. Changes: Auto-generated resources (regenerated with nested types): - AWS::S3::Bucket - 50+ nested model types (CorsConfiguration, ReplicationConfiguration, LifecycleConfiguration, etc.) Manually written resources (added typed nested models): - AWS::OpenSearchService::Domain - ClusterConfig, EBSOptions, VPCOptions, AdvancedSecurityOptions, etc. - AWS::Elasticsearch::Domain - Same nested types as OpenSearch - AWS::EC2::VPCEndpoint - DnsOptions, Tags - AWS::IAM::User - Tags (using existing Tag type) Field type improvements: - TopicARN: ResolvableStr (was ResolvableGeneric) - KMSMasterKeyId: ResolvableStr (was ResolvableGeneric) - UserPoolClientId: ResolvableStr (was ResolvableGeneric) - CertificateArn: ResolvableStr (was ResolvableGeneric) - Various JSON policies: Resolvable[dict] (was ResolvableGeneric) Only PolicyDocument fields remain as ResolvableGeneric since they are intentionally arbitrary JSON (IAM policy documents). Updated tests to work with the new typed access patterns: - LogPublishingOptions and AdvancedOptions now use dict access instead of attribute access Co-Authored-By: Claude Opus 4.5 <[email protected]>
1. Generator reuses existing pycfmodel types (Tag): - Added REUSE_EXISTING_TYPES mapping in CodeGenerator - Tag now imports from pycfmodel.model.resources.properties.tag - Avoids generating duplicate Tag classes in each resource 2. Fixed IAM User LoginProfile typing: - Added proper LoginProfile model with Password and PasswordResetRequired - Updated has_hardcoded_credentials() to work with typed model - LoginProfile is now Optional[ResolvableLoginProfile] 3. Fixed ResolvableModel type: - Removed None from the Union (Optional is applied at field level) - Cleaner type definition: Union[model_cls, FunctionDict] 4. Added comprehensive tests for ResolvableModel: - test_resolvable_model_accepts_dict - test_resolvable_model_accepts_model_instance - test_resolvable_model_accepts_ref/fn_sub/fn_if - test_resolvable_model_accepts_none_for_optional - test_resolvable_model_rejects_invalid_dict/type - test_resolvable_model_serialization (with and without intrinsics) 5. Generator no longer uses ResolvableGeneric: - All fallbacks now use Resolvable[dict] instead - Cleaner generated code without Generic import Co-Authored-By: Claude Opus 4.5 <[email protected]>
1. Replaced global variable caching with SchemaRegistry class: - Encapsulates schema downloading and caching logic - Better testability and cleaner separation of concerns - Supports per-region caching 2. Extracted test fixture helper function: - Added generate_and_load_module() context manager - Eliminates code duplication between test fixtures - Proper cleanup with try/finally pattern 3. Removed unused Optional import Co-Authored-By: Claude Opus 4.5 <[email protected]>
Added tests for 10 resource types that had model implementations but no corresponding test files: - test_autoscaling_auto_scaling_group.py - test_cloudwatch_alarm.py - test_dynamodb_table.py - test_elbv2_listener.py - test_elbv2_target_group.py - test_iam_instance_profile.py - test_route53_record_set.py - test_sns_subscription.py - test_sns_topic.py - test_sqs_queue.py Each test file covers: - Basic resource instantiation - Property validation - Required field validation - Nested object handling - Common use cases for the resource type Co-Authored-By: Claude Opus 4.5 <[email protected]>
173f5d4 to
4ed7aad
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a CLI script that generates static pycfmodel resource classes from AWS CloudFormation schemas, plus the top 10 most common unmodeled resource types (increasing coverage from 51.9% to 71.8%).
Usage
New Resource Types Added
Based on analysis of 2,211 production CloudFormation templates containing 5,285 resources:
Coverage improvement: 51.9% → 71.8% (1,051 additional resources now properly typed)
Generator Features
ResolvableStr,ResolvableGeneric, etc.)Resourcebase class--dry-runoption to preview generated code--list-typesoption to list all 1510 available AWS resource typesExample Generated Code
Test plan
🤖 Generated with Claude Code