Senior Data Engineer
Creative Commons
Creative Commons is working on a project to index every piece of content that's openly licensed online and making it searchable and discoverable through CC Search and the CC Catalog API. The Senior Data Engineer reports to the Director of Engineering and is responsible for building and maintaining the data that powers those products (the CC Catalog). This project will unite billions of records for openly-licensed and public domain works and metadata, across multiple platforms, diverse media types, and a variety of user communities and partners. All the code we write is open source and we’re a 100% remote team.
Primary responsibilities
- Architect, build, and maintain the existing CC Catalog, including:
- Ingesting content from new and existing sources of CC-licensed and public domain works.
- Scaling the catalog to support billions of records and various media types.
- Implementing resilient, distributed data solutions that operate robustly at web scale.
- Automating data pipelines and workflows.
- Collaborating with the Backend Software Engineer and Front End Engineer to support the smooth operation of the CC Catalog API and CC Search.
- Restructuring the code and workflows such that it allows community contributors to identify new sources of content and add new data to the catalog.
- Guiding new contributors and potentially participating in projects such as Google Summer of Code as a mentor.
- Writing blog posts, maintaining documentation, reviewing pull requests, and responding to issues from the community.
Qualifications and requirements
- Demonstrated experience building and deploying large scale data services, including database design and modeling, ETL processing, and performance optimization
- Proficiency with Python
- Proficiency with Apache Spark or similar tools
- Experience with cloud computing platforms (AWS or similar)
- Experience with Apache Airflow or other workflow management software
- Experience with machine learning or interest in picking it up
- Fluent in English
- Excellent written and verbal communication skills
- Ability to work independently, build good working relationships and actively communicate, contribute, and speak up in a remote work structure
- Curiosity and a desire to keep learning
- Commitment to consumer privacy and security
- Nice to have (but not required):
- Experience with contributing to or maintaining open source software
- Experience with web crawling
- Experience with Docker
Diversity & inclusion
We believe that diverse teams build better organizations and better services. Applications from qualified candidates from all backgrounds, including those from under-represented communities, are very welcome. Creative Commons works openly as part of a global community, guided by collaboratively developed codes of conduct and anti-harassment policies.
Work environment and location
Creative Commons is a fully-distributed organization — we have no central office. This position is in a remote working environment and you can be anywhere in the world as long as you’re available for meetings between 2 PM to 8 PM UTC. You must have reasonable mobility for travel to twice-annual all-staff meetings and the CC Global Summit (a total of 3 trips per year). We provide a subsidy towards high-speed broadband access. Laptop/desktop computer and necessary resources are supplied.
Salary and benefits
Creative Commons is a leading non-profit employer, offering competitive salaries and benefits, including health and wellness plans, annual retirement contributions, and a positive, supportive work environment. We offer competitive salary in the range for this position from $100,000 - $120,000 USD, commensurate with relevant skills, experience, and location.
How to apply
Please email your cover letter and resume as a single PDF to “jobs@creativecommons.org” with the subject heading of “Data Engineer / [Last Name].” Phone calls and messages will not be returned.
Similar Jobs
