I walk through the process of determining the best way to build a document management and note keeping system based on customer input and requirements.
Based upon reliability and availability and costing inputs, the choices are winnowed down. Growth and additional reliability may come into play later
but need to be accounted for now and options addressed.
Initial Requirements
In AWS environment I would like to create an app that manages files and documents that can go from small to about ‘medium’ sized… say about 2MB…
probably text and pdf for the most part… maybe some pictures, videos and audio.
I would want to use some database to manage the documents,
information like when the docs were created and added, and also to manage tags…
tagging the files would give me the ability to do directed, keyword type searches in order to find these documents.
The types of queries and searches will not be known when or as the documents are created… so the searches would be more ad hoc, again using the tagging that I will create and manage.
I figure there are at least several ways of architecting this. I could keep all the information within databases maintained withing AWS (so perhaps Postgres or MySQL) but would no-sql be an option when you consider the other factors?
These other factors would be based around performance and costs (the infrastructure and the operating costs). Keeping the data within S3 and allowing the database to manage the location, and retrieval of this information via pointers and reference seem like a way to keep the database content smaller?
What other questions should I be asking?
And, what would be a good overall architecture for this?
Initial Discussion
To architect an AWS-based application that manages and searches through documents with considerations for performance, cost, and flexibility, I can explore multiple options and considerations.
Here’s an overview of key factors, potential questions, and a recommended initial architecture:
Key Questions to Ask:
Data Volume and Growth: How much data are you expecting initially, and
what is the expected growth rate?
Query Patterns: What types of queries will be run?
Will they be predominantly metadata-based, or will you need full-text search capabilities?
Data Consistency: How critical is data consistency for your application?
Can eventual consistency suffice?
Scalability: How scalable does the solution need to be?
Are there peak times when the load will be significantly higher?
Cost Sensitivity: What is your budget for storage and compute?
How sensitive are you to ongoing operational costs?
Access Patterns: How often will documents be read versus written or updated?
Compliance and Security: Are there specific compliance or security requirements for storing and accessing the documents?
Initial Suggested Architecture:
Storage: Amazon S3
Purpose: Store the actual documents (text, PDF, images, videos, audio).
Benefits:
Cost-effective for large amounts of data.
Highly durable and scalable.
Easy to integrate with other AWS services.
Considerations:
Use lifecycle policies to transition old data to infrequent access or archive classes to save costs.
Metadata and Tags Management: Amazon DynamoDB or Amazon RDS
DynamoDB (NoSQL):
Use Case: If your access patterns are well-defined and you’re primarily querying by specific attributes like tags, creation dates, etc.
Benefits:
Fully managed, highly scalable, and fast for key-value and document-based access.
Flexible schema for varying document metadata.
Low operational overhead.
Considerations: Ideal for applications that require scalability and have predictable query patterns.
RDS (PostgreSQL/MySQL):
Use Case: If you need complex querying, joins, and strong transactional consistency.
Benefits:
Relational model suitable for ad hoc queries.
Supports structured data and complex relationships.
Considerations: Requires more management effort for scaling and performance tuning.
Search Capabilities: Amazon OpenSearch Service (formerly Elasticsearch)
Purpose: Provide full-text search capabilities, including keyword and ad hoc queries.
Benefits:
Designed for complex search and analytics use cases.
Supports indexing of document metadata and contents if needed.
Enables fast and flexible querying of tags and metadata.
Considerations: Costs can increase with the amount of data indexed and search complexity.
Serverless or Managed Compute: AWS Lambda or Amazon EC2
AWS Lambda:
Use Case: For event-driven processing, like tagging documents on upload or handling search queries.
Use Case: If you need long-running processes or custom environments.
Benefits: More control over the computing environment.
Indexing and Search Integration:
On document upload to S3, use AWS Lambda to automatically extract metadata, index it in DynamoDB or RDS, and update the search index in OpenSearch.
Security and Compliance:
Use AWS Identity and Access Management (IAM) for fine-grained access control.
Enable S3 bucket policies and encryption at rest (S3 Server-Side Encryption).
Consider AWS Key Management Service (KMS) for managing encryption keys.
Monitoring and Logging:
Use Amazon CloudWatch for monitoring and logging to ensure performance and operational health.
Enable AWS CloudTrail for auditing access to S3 and other services.
Final Considerations:
Cost Analysis: Regularly analyze costs for S3 storage, DynamoDB or RDS usage, and OpenSearch queries.
Performance Tuning: Continuously optimize database queries, index configurations, and storage class policies based on usage patterns.
Backup and Disaster Recovery: Implement backups for RDS and snapshots for DynamoDB. Use versioning in S3 for document recovery.
By combining these AWS services, you can build a robust, scalable, and cost-effective document management system tailored to your specific needs.