Onboarding

In order to run all projects and solutions that constitute KakeiBro as a whole, we need to have installed different tools in our system.

Philosophy

Being software a constantly changing and evolving industry, we should always aim at updating both our tools and technologies, on top of our overall knowledge. Hence tools and neccesary programs should be kept up to date following both STS and LTS release lifecycles. Since the stack is based around .NET, Node and Redis. We should always be mindful of their lifecycles:

The main reason as to always stay up to date is for security reasons, but also due to the fact that new features are always welcome, specially if they make previous problems that we had experienced or are currently experiencing way easier to solve.

Note: As per Node’s release docs, prod apps should only use Active LTS or Maintenance LTS releases.

Neccesary Tools

  • Git

  • .NET 9 Runtime and SDK

  • nvm, (windows variant: nvm-windows)

  • pnpm

  • Docker (on Windows you will have to leverage WSL and Docker Desktop)

  • Visual Studio Code, (or its truly FOSS fork VSCodium)

    • IMPORTANT: It’s highly advisable to use this IDE since the project setup has been catered towards a workstation with this specific IDE.

    • Other options: Neovim, WebStorm

  • JetBrains Rider

    • IMPORTANT: Highly recommended IDE for a seamless and rich development experience.

    • Visual Studio 2022

      • It’s recommended to have this installed just in case an error or some really obscure functionality should be leveraged when it comes to working with the project. (A quick jump into it, running and getting out type of flow).

  • Cake

    • NOTE: You don’t have to manually install it, since there’s a script file at the root of the repository that will take care of installing it.

  • gcloud CLI

  • jq, a CLI tool that helps us deal with JSON and files, this is key for our Git Hooks setup since we need to pretty format files after sanitization of sensitive credentials

    • NOTE: On Linux one can easily get it from any package manager, on windows it’s recommended to install it through choco, choco install jq

CI/CD Pillars

Based on theory and ideas from DevOps practices, we can extrapolate different pillars or best practices that we should aim at having on a modern software solution, a solution that should ensure efficient, automated software development and deployment. Each general principle has other ideas that could go as deep as becoming as the main pillar depending on perspective:

Automation:

  • Build Automation: We should automate the process of compiling code and creating artifacts.

  • Test Automation: We should automate tests, unit, integration and even E2E to ensure code quality and catching of bugs early in the development process.

  • Deployment Automation: We should automate the process of deploying code to various environments.

  • Infrastructure as Code (IaC): We should manage infrastructure through code to ensure consistency and reproducibility.

Continous Integration:

  • Frequent Code Integration: Developers should merge their code changes into a shared repository, often multiple times a day.

  • Automated Builds: Every code change triggers an automated build process to compile the code and create executable artifacts.

  • Automated Testing: When automating tests we are also ensuring that the code works as expected and doesn’t introduce regressions.

  • Fast Feedback: Developers receive immediate feedback on the success or failure of their changes, enabling quick fixes.

Continous Delivery:

  • Deployment Pipeline: Automated pipelines are used to build, test, and prepare code for release to production.

  • Environment Consistency: Code is deployed to staging or pre-production environments that closely mirror the production environment.

  • Manual Approval Gates: Releases to production are often gated by manual approvals ensuring human oversight before deployment.

  • Reliable Releases: The goal is to ensure that software is always in a releasable state, even if deployments to production are not fully automated.

Monitoring and Logging:

  • Real-Time Monitoring: Monitoring applications and infrastructure in real-time to detect issues.

  • Logging and Metrics: Collecting logs and metrics to analyze performance and troubleshoot problems.

  • Feedback Loops: Using feedback from monitoring and users to improve the software and CI/CD process.

Collaboration and Communication:

  • Cross-Functional Teams: Encouraging collaboration between development, operations, and QA teams.

  • Shared Responsibility: Promoting a culture where everyone is responsible for the quality and reliability of the software.

  • Continuous Improvement: Regularly reviewing and improving the CI/CD process to adapt to changing needs.

Version Control:

  • Single Source of Truth: All code and configuration files are stored in a version control system (e.g., Git).

  • Branching Strategies: Using strategies like GitFlow, trunk-base development, or feature branching to manage code changes.

  • Code Reviews: Ensuring peer reviews and pull requests to maintain code quality.

Infrastructure as Code (IaC):

  • Infrastructure managed through code: Infrastructure should be architected and run with tools and frameworks that allow for it to be version controlled, by making infrastructure provisioning through machine-readable definition files, rather than manual processes (e.g., Terraform, Ansible, Azure Bicep/ARM).

  • Idempotency: When having infrastructure as code, it enables for it to be repeatable, scalable and trackeable. (e.g: If some infra change broke something, we can roll-back, or identify a point in which a failure took place).

  • Testability: CI/CD pipelines can also be used against the IaC code-base, and since everything is declarable and can be implemented like pieces of code, management, reusability and trust is improved.

Security and Compliance:

  • CI/CD Pipelines: Through the usage of tools we should be running security and vulnerability checks in CI/CD Pipelines.

  • Static Analysis: Tools that we run against our code-base should be a combination of static analysis, vulnerability scanning and compliance checks.

  • Secrets Management: We should manage and inject secrets into the CI/CD pipeline with tools such as HashiCorp Vault or GitHub Secrets.

KakeiBro’s Pillars

From everything laid out at CI/CD Pillars, we will extrapolate just enough for the project, and establish the specific ways we are putting into practice the theory.

We are focusing on three most important ideas:

  1. Automation and Efficiency: We are working towards speed, reliability and consistency by automating as much of the process as we can, code integration, testing, builds, deployments, security checks. Lastly, we will leverage version control and declarative configurations to maintain a single source of truth for both application code and some level of the infrastructure (artifacts in the form of docker images).

    This is the main reason as to why we are putting so much emphasis in Frontend and Backend hooks that use tools to keep in check both that the code-base adheres to code conventions, but also that it builds, and that tests pass, and it’s through config files on each respective repository that we can shape the way tools will police our code everytime we try to integrate (make a commit, push to version control, after the push is made, or a PR is put up to merge into master).

  2. Security and Compliance at every stage: We will embed a certain level of security best practices into our CI/CD pipelines so that it checks for outdated dependencies and perhaps vulnerable dependencies so that we are alerted to update. It’s also through the usage of these automated tools that we will avoid pushing sensitive information into our repository, on top of making sure that we obfuscate sensitive information by using GitHub Secrets and keeping in check what information should be kept outside of the public eye.

  3. Collaboration, Monitoring, and Continous Improvement: It’s due to the automation and culture that is built around the whole project that we are touching different verticals of a typical organization, the idea is to always collaborate across different areas, and share the responsibility of the quality and reliability. Which means that with real-time monitoring, feedback loops, and iterative processes we can both improve the software and the delivery pipeline, this isn’t just a static process that once it’s up it will never change, we should always recognize where a step is not bringing that much value and update it, or just remove it. Plus bringing in other steps or workflows that should make the sofware and the process of developing it even better.

Github Actions

It’s around the CI/CD Pillars that GitHub has its own set of tools that help with automation, and that is in the form of Github Actions, the big advantage (and also disadvantage) is that they have built-in abstractions. There are tons and tons of already pre-built chunks of code that take care of a specific process that’s a common use case when building CI/CD Pipelines, e.g:

  • actions/checkout@v4 = Call to automatically run logic that will clone the repository to which the Action might be attached to, it also runs clean up logic by the end of the flow.

  • actions/cache@v4 = Call to automatically run logic that will attempt at retrieving data from a nebulous cloud instance and restore in the current build machine’s context, it will also cache by the end of the whole flow the specific paths and all the files there in the same nebulous cloud instance.

  • docker/build-push-action@v4 = Call to automatically run logic that takes care of running docker so that it builds an image based on different parameters, tag it and then push it to a specific repository.

Making usage of these pre-built recipes also helps with more metadata and info that gets outputted when an Action is done. You can get summaries on the build steps, how much was cached, time it took to run specific steps and so on.

You can go and search for each of these actions code in Github under repos since most (if not all) of them are Open Source. This is a great way to keep track of new versions for them as well.

Besides that, we can also inject our own custom logic as we would with any Pipeline platform, but it’s the way we can consume already coded pieces of code that makes our yml files more clean and lean. It’s best to make use of the built-in abstractions since they follow best practices and do more than it seems, both before and after the whole flow, of course, if the need to build a custom type of logic arises, we can jump in and code our own scripts catered to our own specific needs.