Get Started¶

In software development, where efficiency, consistency, and reliability are paramount, automation tools play a crucial role. Makim, an innovative open-source tool, steps into the spotlight to improve automation workflows. It simplifies script execution, environment management, and task dependencies, positioning itself as a great asset in modern development environments.

Introducing Makim¶

Makim elevates project automation by offering a structured, yet flexible approach to manage routine tasks, complex task dependencies, and environment configurations. Its design is centered around the .makim.yaml configuration file, allowing developers to orchestrate their workflows with precision and ease. Unlike traditional script execution tools, Makim's Python-based architecture and support for multiple programming languages and shells enhance its versatility and applicability across diverse projects.

Especially suited for DevOps Engineers and Software Developers, Makim eliminates redundancy in automation tasks. Its core functionality extends beyond simple script execution, encompassing:

• Argument definition for scripts
• Organization of tasks into groups
• Advanced dependency management between tasks
• Utilization of environment variables and custom variables
• Dynamic content generation with Jinja2 templates
• Specification of working directories for tasks
• Execution flexibility through support for multiple interpreters or shells

Despite its broad capabilities, Makim currently lacks support for Windows but plans to extend its compatibility in future versions.

Getting Started with Makim¶

Installation¶

Makim can be installed via pip or conda, catering to different setup preferences:

• To install Makim using pip, run:

pip install -q "makim==1.16.0"

• For those who prefer conda, execute:

  conda install "makim==1.16.0"
  

Given Makim's active development, pinning to a specific version is recommended to ensure consistency.

For this tutorial, we will disable the output color feature provided by typer, the command-line interface engine used by Makim.

In [2]:

import os
os.environ["NO_COLOR"] = "1"

import os os.environ["NO_COLOR"] = "1"

Configuring .makim.yaml¶

The .makim.yaml file is the foundation of your Makim configuration. Here's how to start:

1. Create the .makim.yaml File: Place this file at the root of your project directory.

2. Define Your Automation Tasks: Configure your tasks, specifying actions, arguments, and dependencies. For example:

In [3]:

%%writefile .makim.yaml
version: 1.0.0
groups:
  clean:
    env-file: .env
    tasks:
      tmp:
        help: Use this task to clean up temporary files
        run: |
          echo "Cleaning up..."
  tests:
    tasks:
     unit:
       help: Build the program
       args:
         clean:
           type: bool
           action: store_true
           help: if not set, the clean dependency will not be triggered.
       hooks:
         pre-run:
          - task: clean.tmp
            if: ${{ args.clean == true }}
       run: |
         echo "Runnint unit tests..."

%%writefile .makim.yaml version: 1.0.0 groups: clean: env-file: .env tasks: tmp: help: Use this task to clean up temporary files run: | echo "Cleaning up..." tests: tasks: unit: help: Build the program args: clean: type: bool action: store_true help: if not set, the clean dependency will not be triggered. hooks: pre-run: - task: clean.tmp if: ${{ args.clean == true }} run: | echo "Runnint unit tests..."

Overwriting .makim.yaml

This setup demonstrates Makim's ability to manage tasks with conditional logic and dependencies.

Exploring Makim's CLI¶

Makim's CLI provides insights into available commands, arguments, and configurations through the auto-generated help menu:

In [4]:

!makim --help

!makim --help

                                                                                
 Usage: makim [OPTIONS] COMMAND [ARGS]...                                       
                                                                                
 Makim is a tool that helps you to organize and simplify your helper commands.  
                                                                                
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --version             -v            Show the version and exit                │
│ --file                        TEXT  Makim config file [default: .makim.yaml] │
│ --dry-run                           Execute the command in dry mode          │
│ --verbose                           Execute the command in verbose mode      │
│ --install-completion                Install completion for the current       │
│                                     shell.                                   │
│ --show-completion                   Show completion for the current shell,   │
│                                     to copy it or customize the              │
│                                     installation.                            │
│ --help                              Show this message and exit.              │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────╮
│ clean.tmp        Use this task to clean up temporary files                 │
│ tests.unit       Build the program                                           │
╰──────────────────────────────────────────────────────────────────────────────╯
                                                                                
 If you have any problem, open an issue at:                                     
 https://github.com/osl-incubator/makim                                         
                                                                                

This feature facilitates easy access to Makim's functionalities, enhancing usability and understanding of the tool.

Executing Your First Commands¶

With your .makim.yaml file set up, you can begin to use makim:

In [5]:

!makim clean.tmp

!makim clean.tmp

Makim file: .makim.yaml
Cleaning up...

In [6]:

!makim tests.unit

!makim tests.unit

Makim file: .makim.yaml
Runnint unit tests...

In the case you type your command wrong, Makim will suggest you some alternative:

In [7]:

!makim tests.unittest

!makim tests.unittest

Usage: makim [OPTIONS] COMMAND [ARGS]...
Try 'makim --help' for help.
╭─ Error ──────────────────────────────────────────────────────────────────────╮
│ No such command 'tests.unittest'.                                            │
╰──────────────────────────────────────────────────────────────────────────────╯
Command tests.unittest not found. Did you mean tests.unit'?

Makim CLI is empowered by Typer, and it allows us to have auto-completion for Makim groups and tasks! If you want to install, you can run the following command:

In [8]:

!makim --install-completion

!makim --install-completion

bash completion installed in /home/xmn/.bash_completions/makim.sh
Completion will take effect once you restart the terminal

After this command you will need to restart the terminal in order to use this auto-completion feature.

Advanced Features and Examples¶

Makim's adaptability is showcased through various features and practical examples:

• Conditional Dependencies and Arguments: Define complex task dependencies with conditional execution based on passed arguments.
• Dynamic Configuration with Jinja2: Leverage Jinja2 templates for advanced scripting and dynamic content generation.
• Environment and Custom Variable Management: Organize and utilize variables effectively across different scopes of your project.
• Specifying Working Directories: Control the execution context of your tasks by setting working directories.

These examples underscore Makim's capability to accommodate intricate automation scenarios, streamlining development workflows.

Exploring Makim Through Examples¶

Utilizing Various Interpreters¶

Makim extends its functionality beyond conventional script execution by supporting various interpreters and shell languages, facilitating a versatile development environment. While xonsh is the default interpreter - blending the capabilities of Bash and Python for an enriched command-line experience - Makim's architecture allows for seamless integration with other environments. For developers seeking to leverage this feature, a foundational understanding of xonsh can be beneficial. Comprehensive details and usage guidelines are available in the official xonsh documentation.

This section demonstrates executing straightforward commands across multiple interpreters, showcasing Makim's adaptability to diverse programming contexts.

In [9]:

%%writefile .env
MSG_PREFIX="Running Makim: Hello, World,"

%%writefile .env MSG_PREFIX="Running Makim: Hello, World,"

Overwriting .env

In [10]:

%%writefile .makim.yaml
version: 1.0
env-file: .env
groups:
  tests:
    tasks:
      node:
        help: Test using nodejs
        backend: node
        run: console.log("${{ env.MSG_PREFIX }} from NodeJS!");
      perl:
        help: Test using perl
        backend: perl
        run: print "${{ env.MSG_PREFIX }} from Perl!\n";

      python:
        help: Test using php
        baackend: python
        run: print("${{ env.MSG_PREFIX }} from Python!")

      r:
        help: Test using R
        backend: Rscript
        run: print("${{ env.MSG_PREFIX }} from R!")

      sh:
        help: Test using sh
        backend: sh
        run: echo "${{ env.MSG_PREFIX }} from sh!"

      run-all:
        help: Run tests for all the other tasks
        hooks:
          pre-run:
            - task: node
            - task: perl
            - task: python
            - task: r
            - task: sh

%%writefile .makim.yaml version: 1.0 env-file: .env groups: tests: tasks: node: help: Test using nodejs backend: node run: console.log("${{ env.MSG_PREFIX }} from NodeJS!"); perl: help: Test using perl backend: perl run: print "${{ env.MSG_PREFIX }} from Perl!\n"; python: help: Test using php baackend: python run: print("${{ env.MSG_PREFIX }} from Python!") r: help: Test using R backend: Rscript run: print("${{ env.MSG_PREFIX }} from R!") sh: help: Test using sh backend: sh run: echo "${{ env.MSG_PREFIX }} from sh!" run-all: help: Run tests for all the other tasks hooks: pre-run: - task: node - task: perl - task: python - task: r - task: sh

Overwriting .makim.yaml

In [11]:

!makim --help

!makim --help

                                                                                
 Usage: makim [OPTIONS] COMMAND [ARGS]...                                       
                                                                                
 Makim is a tool that helps you to organize and simplify your helper commands.  
                                                                                
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --version             -v            Show the version and exit                │
│ --file                        TEXT  Makim config file [default: .makim.yaml] │
│ --dry-run                           Execute the command in dry mode          │
│ --verbose                           Execute the command in verbose mode      │
│ --install-completion                Install completion for the current       │
│                                     shell.                                   │
│ --show-completion                   Show completion for the current shell,   │
│                                     to copy it or customize the              │
│                                     installation.                            │
│ --help                              Show this message and exit.              │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────╮
│ tests.node             Test using nodejs                                     │
│ tests.perl             Test using perl                                       │
│ tests.python           Test using php                                        │
│ tests.r                Test using R                                          │
│ tests.run-all          Run tests for all the other tasks                   │
│ tests.sh               Test using sh                                         │
╰──────────────────────────────────────────────────────────────────────────────╯
                                                                                
 If you have any problem, open an issue at:                                     
 https://github.com/osl-incubator/makim                                         
                                                                                

Prior to executing these tasks, it is necessary to install the required dependencies:

In [12]:

!mamba install -q -y perl nodejs r-base sh

!mamba install -q -y perl nodejs r-base sh

Proceed to execute all defined tasks by invoking the run-all task, which encapsulates all other tasks as its dependencies for a sequential execution process:

In [13]:

!makim tests.run-all

!makim tests.run-all

Makim file: .makim.yaml
Running Makim: Hello, World, from NodeJS!
(node:1634785) Warning: The 'NO_COLOR' env is ignored due to the 'FORCE_COLOR' env being set.
(Use `node --trace-warnings ...` to show where the warning was created)
Running Makim: Hello, World, from Perl!
Running Makim: Hello, World, from Python!
[1] "Running Makim: Hello, World, from R!"
Running Makim: Hello, World, from sh!

In scenarios where your chosen interpreter supports debugging - such as Python or Xonsh through the use of breakpoint() - you can introduce a breakpoint within your code. This enables the debugging of your Makim task, allowing for an interactive examination of the execution flow and variable states.

Using Variables (vars)¶

Makim facilitates the definition of variables within the .makim.yaml configuration, supporting all the YAML data types, including strings, lists, and dictionaries. This feature enhances script configurability and reusability across different tasks and environments.

Consider reviewing the provided example to understand how to effectively leverage variables in your Makim configurations:

In [14]:

%%writefile .makim.yaml
version: 1.0

vars:
  project-name: "my-project"
  dependencies:
    "dep1": "v1"
    "dep2": "v1.1"
    "dep3": "v2.3"
  authorized-users:
    - admin1
    - admin2
    - admin3

groups:
  staging:
    vars:
      env-name: "staging"
      staging-dependencies:
        "dep4": "v4.3"
        "dep5": "v1.1.1"
      staging-authorized-users:
        - staging1
        - staging2
        - staging3
    tasks:
      create-users:
        help: Create users for staging, this example uses jinja2 for loop.
        # each task can also specify their `vars`, but it will not be used in this example
        run: |
          def create_user(username):
              print(f">>> creating user: {username} ... DONE!")
                                                                
          print("create admin users:")
          {% for user in vars.authorized_users %}
          create_user("${{ user }}")
          {% endfor %}

          print("\ncreate staging users:")
          {% for user in vars.staging_authorized_users %}
          create_user("${{ user }}")
          {% endfor %}

      install:
        help: install deps for staging using native xonsh `for` loop (it could work with Python as well)
        # each task can also specify their `vars`, but it will not be used in this example
        run: |
          def install(package, version):
              print(f">>> installing: {package}@{version} ... DONE!")
                                            
          print("install global dependencies:")
          for package, version in ${{ vars.dependencies | safe }}.items():
              install(package, version)

          print("\ninstall staging dependencies:")
          for package, version in ${{ vars.staging_dependencies | safe }}.items():
              install(package, version)

%%writefile .makim.yaml version: 1.0 vars: project-name: "my-project" dependencies: "dep1": "v1" "dep2": "v1.1" "dep3": "v2.3" authorized-users: - admin1 - admin2 - admin3 groups: staging: vars: env-name: "staging" staging-dependencies: "dep4": "v4.3" "dep5": "v1.1.1" staging-authorized-users: - staging1 - staging2 - staging3 tasks: create-users: help: Create users for staging, this example uses jinja2 for loop. # each task can also specify their `vars`, but it will not be used in this example run: | def create_user(username): print(f">>> creating user: {username} ... DONE!") print("create admin users:") {% for user in vars.authorized_users %} create_user("${{ user }}") {% endfor %} print("\ncreate staging users:") {% for user in vars.staging_authorized_users %} create_user("${{ user }}") {% endfor %} install: help: install deps for staging using native xonsh `for` loop (it could work with Python as well) # each task can also specify their `vars`, but it will not be used in this example run: | def install(package, version): print(f">>> installing: {package}@{version} ... DONE!") print("install global dependencies:") for package, version in ${{ vars.dependencies | safe }}.items(): install(package, version) print("\ninstall staging dependencies:") for package, version in ${{ vars.staging_dependencies | safe }}.items(): install(package, version)

Overwriting .makim.yaml

Now, let's proceed to create users within the staging environment:

In [15]:

!makim staging.create-users

!makim staging.create-users

Makim file: .makim.yaml
create admin users:
>>> creating user: admin1 ... DONE!
>>> creating user: admin2 ... DONE!
>>> creating user: admin3 ... DONE!

create staging users:
>>> creating user: staging1 ... DONE!
>>> creating user: staging2 ... DONE!
>>> creating user: staging3 ... DONE!

Last but not least, let's run the install task:

In [16]:

!makim staging.install

!makim staging.install

Makim file: .makim.yaml
install global dependencies:
>>> installing: dep1@v1 ... DONE!
>>> installing: dep2@v1.1 ... DONE!
>>> installing: dep3@v2.3 ... DONE!

install staging dependencies:
>>> installing: dep4@v4.3 ... DONE!
>>> installing: dep5@v1.1.1 ... DONE!

Defining Arguments¶

Makim enhances script flexibility by allowing the use of arguments. It enables not only the definition of arguments for tasks but also the passing of arguments to dependencies and the specification of conditions for those dependencies.

Explore this functionality through this example:

In [17]:

%%writefile .makim.yaml
version: 1.0.0
groups:
  print:
    env-file: .env
    tasks:
      name:
        help: Print given name
        args:
          name:
            type: str
            required: true
        run: print("${{ args.name }}")
      list:
        help: Build the program
        args:
          i-am-sure:
            type: bool
        hokos:
          pre-run:
            - task: print.name
              if: ${{ args.i_am_sure == true }}
              args:
                name: Mary
            - task: print.name
              if: ${{ args.i_am_sure == true }}
              args:
                name: John
            - task: print.name
              if: ${{ args.i_am_sure == true }}
              args:
                name: Ellen
            - task: print.name
              if: ${{ args.i_am_sure == true }}
              args:
                name: Marc

%%writefile .makim.yaml version: 1.0.0 groups: print: env-file: .env tasks: name: help: Print given name args: name: type: str required: true run: print("${{ args.name }}") list: help: Build the program args: i-am-sure: type: bool hokos: pre-run: - task: print.name if: ${{ args.i_am_sure == true }} args: name: Mary - task: print.name if: ${{ args.i_am_sure == true }} args: name: John - task: print.name if: ${{ args.i_am_sure == true }} args: name: Ellen - task: print.name if: ${{ args.i_am_sure == true }} args: name: Marc

Overwriting .makim.yaml

In [18]:

!makim print.list

!makim print.list

Makim file: .makim.yaml

In [19]:

!makim print.list --i-am-sure

!makim print.list --i-am-sure

Makim file: .makim.yaml
Mary
John
Ellen
Marc

Utilizing Environment Variables¶

The previous sections demonstrated the use of environment variables. Here, we'll delve into their application in more detail.

Makim permits the incorporation of environment variables from .env files or directly within the .makim.yaml file, applicable at global, group, and task levels.

Examine an example to understand the implementation:

In [20]:

%%writefile .env
ENV=dev

%%writefile .env ENV=dev

Overwriting .env

In [21]:

%%writefile .makim.yaml
version: 1.0
env-file: .env
env:
  GLOBAL_VAR: "1"
groups:
  global-scope:
    env:
      GROUP_VAR: "2"
    tasks:
      test-var-env-file:
        help: Test env variable defined in the global scope from env-file
        run: |
          import os
          assert str(os.getenv("ENV")) == "dev"

      test-var-env:
        help: Test env variable defined in the global scope in `env` section
        env:
          TARGET_VAR: "3"
        run: |
          import os
          # you can get an environment variable directly with xonsh/python
          assert str(os.getenv("GLOBAL_VAR")) == "1"
          # or you can get an environment variable using jinja2 tag
          assert "${{ env.GROUP_VAR }}" == "2"
          assert "${{ env.get("TARGET_VAR") }}" == "3"
          assert "${{ env.get("UNKNOWN_TARGET_VAR", "4") }}" == "4"

%%writefile .makim.yaml version: 1.0 env-file: .env env: GLOBAL_VAR: "1" groups: global-scope: env: GROUP_VAR: "2" tasks: test-var-env-file: help: Test env variable defined in the global scope from env-file run: | import os assert str(os.getenv("ENV")) == "dev" test-var-env: help: Test env variable defined in the global scope in `env` section env: TARGET_VAR: "3" run: | import os # you can get an environment variable directly with xonsh/python assert str(os.getenv("GLOBAL_VAR")) == "1" # or you can get an environment variable using jinja2 tag assert "${{ env.GROUP_VAR }}" == "2" assert "${{ env.get("TARGET_VAR") }}" == "3" assert "${{ env.get("UNKNOWN_TARGET_VAR", "4") }}" == "4"

Overwriting .makim.yaml

In [22]:

!makim global-scope.test-var-env-file

!makim global-scope.test-var-env-file

Makim file: .makim.yaml

In [23]:

!makim global-scope.test-var-env

!makim global-scope.test-var-env

Makim file: .makim.yaml

Specifying the Working Directory¶

Makim provides the capability to set a specific working directory for tasks at any scope: global, group, or task.

Review a straightforward example to learn how to apply this feature:

In [24]:

%%writefile .makim.yaml
version: 1.0
dir: "/tmp"

groups:
  check-wd:
    tasks:
      is-tmp:
        help: Test if working directory is `tmp`
        run: |
          import os
          print(os.getcwd())
          assert os.getcwd() == "/tmp"

%%writefile .makim.yaml version: 1.0 dir: "/tmp" groups: check-wd: tasks: is-tmp: help: Test if working directory is `tmp` run: | import os print(os.getcwd()) assert os.getcwd() == "/tmp"

Overwriting .makim.yaml

In [25]:

!makim check-wd.is-tmp

!makim check-wd.is-tmp

Makim file: .makim.yaml
/tmp

This tutorial concludes with a showcase of Makim's key features. While this overview covers the essentials, diving deeper into Makim will reveal more advanced and intriguing ways to leverage its capabilities.

Contributing to Makim¶

Makim's growth is propelled by its community. Contributions, whether through code, documentation, or feedback, are welcome. Explore the GitHub repository and consider contributing to foster Makim's development.

Conclusion¶

Makim stands out as a transformative tool in project automation, bridging the gap between simplicity and complexity. Its comprehensive feature set, coupled with the flexibility of its configuration, makes Makim a quintessential tool for developers and DevOps engineers alike. As you incorporate Makim into your projects, its impact on enhancing productivity and consistency will become evident, marking it as an indispensable part of your development toolkit.

Dive deeper into Makim's functionalities by visiting the official documentation. Try it and let us know your thoughts about it!