Senior Data Engineer

GreatAmerica is a highly successful entrepreneurial company providing equipment financing to businesses across the United States. Our exemplary customer service, our principle-centered business philosophy and our team-based operating approach are key to our success and growth. We have also recently become a bank! We are Looking to Add a Key Member to Our Enterprise Technology Group! The Senior Data Engineer is a hands-on technical leader who designs, builds, modernizes, and operates reliable data products and platform capabilities across GreatAmerica’s cloud and hybrid data environment. This role is heavily focused on modernization using Databricks, Python/PySpark, SQL, Azure, and workflow orchestration, while also maintaining and supporting the on-premises footprint of SQL Server and SSIS. The Senior Data Engineer turns complex business and operational needs into scalable pipelines, trusted data models, reusable engineering patterns, and production-ready solutions that support analytics, reporting, integrations, and emerging AI/ML use cases. The role combines delivery and operational ownership. It requires strong engineering judgment, disciplined software-development practices, effective partnership with business and technology teams, and the ability to guide other engineers while remaining deeply hands-on. As a Senior Data Engineer, you will: Major responsibilities of the position are listed below. To perform the job successfully, the individual must be able to execute each essential duty satisfactorily. Design, build, test, deploy, and support batch and near-real-time data pipelines using Databricks, Python/PySpark, Spark SQL, T-SQL, Azure services, and Apache Airflow/Astronomer Develop scalable lakehouse solutions using Delta Lake and medallion-style patterns, including clearly defined raw/bronze, refined/silver, and curated consumption layers Create and maintain dimensional, relational, and other fit-for-purpose data models that support reporting, analytics, operational integrations, and AI/ML workloads Lead legacy modernization work by assessing existing SQL Server, SSIS, SQL Agent, Informatica PowerCenter/IICS, and custom processes; defining migration patterns; and safely transitioning workloads to strategic platforms Develop, support, and tune on-premises SQL Server data workloads, including databases, stored procedures, SSIS packages, and SQL Agent jobs, ensuring reliable operation and clear documentation while these systems remain in service during the transition Build data quality controls that validate completeness, accuracy, reconciliation, freshness, schema conformance, and source-to-target parity. Investigate discrepancies and drive issues to root cause and resolution Engineer reliable production workflows with idempotency, restartability, retries, backfills, dependency management, alerting, logging, auditability, and clear operational runbooks Apply software-engineering practices to data solutions, including Git-based source control, pull requests, peer review, branching standards, automated testing, CI/CD, environment promotion, and infrastructure as code Optimize pipeline performance and cost through incremental processing, efficient Spark and SQL design, right-sized compute, workload tuning, and elimination of unnecessary full refreshes or data movement Partner with business stakeholders, Business Systems Analysts, application teams, architects, security, infrastructure, governance, analytics, and vendors to translate requirements into maintainable data solutions Design secure data access using Unity Catalog and Azure/Entra patterns, with attention to least privilege, secrets management, lineage, sensitive-data handling, and separation of development, test, and production environments Integrate data from enterprise and third-party platforms, including APIs, files, SQL databases, Salesforce, and other operational systems, while accounting for source-system change and downstream impact Provide technical leadership through design reviews, code reviews, troubleshooting, standards development, reusable frameworks, mentoring, and constructive feedback to engineers at different experience levels Own production outcomes for assigned systems and data products, including incident response, root-cause analysis, corrective actions, documentation, and participation in a 24/7 on-call rotation Contribute actively in an Agile delivery model by refining work, estimating effort, communicating risk and dependencies, and delivering incremental value To be successful in this role you will need: Education Bachelor’s degree in Computer Science, Information Systems, Engineering, or a related field, or an equivalent combination of education and relevant professional experience Experience 7+ years of hands-on data engineering experience building and supporting production data solutions 7+ years of advanced SQL experience, including complex querying, data transformation, performance tuning, and high-volume processing; strong T-SQL experience is required 3+ years of hands-on Python experience for production data engineering 3+ years of hands-on Databricks experience using Apache Spark, Delta Lake, notebooks or jobs in a production environment, including working knowledge of Unity Catalog 3+ years of experience with Microsoft Azure data and platform services such as Azure SQL, Azure Data Lake Storage, Key Vault, Azure Functions, Event Hubs, Azure Monitor, and Microsoft Entra ID Experience developing, supporting, and troubleshooting on-premises SQL Server environments, including SSIS packages and SQL Agent jobs, in a production setting Demonstrated experience designing ETL/ELT pipelines, data warehouses or lakehouses, and dimensional data models Demonstrated experience implementing data quality, reconciliation, automated testing, operational monitoring, alerting, and production support practices Experience with Git-based development, peer review, CI/CD, automated deployments, and environment promotion using Azure DevOps, GitHub Actions, or comparable tooling Experience with Infrastructure as Code (e.g., Terraform or Bicep) to provision and manage data platform resources Experience designing and supporting workflow orchestration with Apache Airflow/Astronomer or a comparable enterprise orchestrator Experience mentoring engineers, leading technical design discussions, and influencing standards without relying on formal supervisory authority Preferred Experience Experience with Databricks Asset Bundles, Lakeflow or declarative pipeline patterns, Databricks SQL, and production administration or performance optimization Experience with data observability and quality frameworks such as DQX, Great Expectations, or comparable approaches Experience with Change Data Capture, Change Tracking, event-driven integration, streaming, and schema-evolution patterns Experience with legacy modernization involving Informatica PowerCenter/IICS, Windows Task Scheduler, or custom integration code Experience supporting governed consumption through Tableau, Excel/Power Query, Salesforce Data 360/Data Cloud, APIs, data sharing, or AI/ML workloads Familiarity with MLOps practices and operationalizing machine learning models, including deployment, monitoring, and supporting feature or training data pipelines Experience using AI-assisted and agentic development tools, such as GitHub Copilot or similar tools, to write, review, test, and troubleshoot code, with sound judgment in validating AI-generated output Experience in financial services or another environment where security, auditability, data lineage, change control, and operational resilience are important Relevant certifications such as Databricks Certified Data Engineer Professional or Microsoft Certified: Azure Data Engineer Associate Skills and Abilities Writes clear, maintainable, testable code and can explain technical trade-offs to both technical and non-technical audiences Balances delivery speed with reliability, security, data quality, maintainability, and cost Works independently on ambiguous problems while engaging the right partners before decisions create downstream risk Communicates production risk, status, dependencies, and incidents directly and with an appropriate sense of urgency Builds effective working relationships across all levels of the organization and with external partners Learns unfamiliar systems quickly and can operate effectively across both modern cloud platforms and legacy environments during transition Produces useful technical documentation, runbooks, diagrams, standards, and implementation guidance Uses AI tools responsibly, validating generated code and protecting sensitive data Sharing rewards is an integral part of our culture. We believe in the value of hard work and reward our employees beyond the paycheck. Our total rewards package is based on eligibility and includes: Financial Benefits Competitive Compensation Monthly Bonuses for Eligible Employees 401(k) and Company Match Annual Profit Sharing Paid Time Off Health, Wellbeing, and Family Planning Benefits Paid Vacation Paid Sick Days Paid Holidays Gym Reimbursement Health Insurance Dental Insurance Vision Insurance Short-Term and Long-Term Disability Company Paid Life Insurance Flexible Spending Accounts (FSA) Health Savings Accounts (HSA) Employee Assistance Program Parental Leave Education and Career Planning Benefits Tuition Assistance Networking Opportunities Leadership Development Opportunities Perks Paid Parking Service Awards Hybrid work arrangements Business casual environment A strong organizational culture focused on our greatest asset: you ! If your experience aligns closely, please apply. We value diverse backgrounds and adding new perspectives. We encourage you to apply if you can make a strong impact in this role at www.greatamerica.com/careers . Please note, applicants must be authorized to work for any employer in the U.S. We are unable to sponsor or take over sponsorship of an employment visa.