Cargo + gitlab + private repo

Hello,

I have a problem with the CI in Git-Lab using cargo when a private crate has a dependency on another private crate.
I have the following report:

Updating git repository https://gitlab.com/hud-software/core/macros.git
[error: failed to load source for a dependency on macros
Caused by:
Unable to update https://gitlab.com/hud-software/core/macros.git
Caused by:
failed to clone into: /usr/local/cargo/git/db/macros-e2e610c29305fea1
Caused by:
failed to authenticate when downloading repository
attempted to find username/password via git's credential.helper support, but failed
Caused by:
failed to acquire username/password from local configuration
ERROR: Job failed: exit code 1

I'm absolutely not familiar with CI on Git-Lab and Docker.
Do you know how I can fix this ?
Thank you

It's an authentication issue. In the same way that I wouldn't be able to git clone your private repository, it looks like the CI runner (when you execute cargo build) isn't able to fetch the repository.

This StackOverflow question may be helpful:

https://stackoverflow.com/questions/25689231/getting-gitlab-ci-to-clone-private-repositories

We use GitLab CI at work and the normal way I'd depend on another private project is via git submodules. So you'd add the macros repository as a git submodule (e.g. under the vendor/ directory) then tell cargo to fetch the dependency from the filesystem by adding macros = { path = "./vendor/macros" }) under [dependencies] in Cargo.toml.

1 Like

Thank you,

I will try with submodule and give a feedback after :wink:

1 Like

So after some holidays and testing CI on GitLab and BitBucket with Cargo for private repositories, I share what I did and how I make it works. ( I'm not a native english speaker I will try to do my best :wink: )

I used SSH to clone repository instead of https because GitLab required to set dependent repository as a git submodule. This is not acceptable in my point of view.

Cargo and SSH consideration:

  1. cargo.toml

    [dependencies]
    macros = { git = "ssh://git@bitbucket.org/A.git" }

  2. .cargo\config

    [net]
    git-fetch-with-cli = true`

    You also need to add in the file .cargo\config to tell cargo the use the git command line interface

  3. Generated SSH key should not have a passphrase.

BitBucket

BitBucket is far more user friendly and simple to setup. Here is the step to follow if repository B depends of repository A.
The pipeline script for A and B are the same :

bitbucket-pipeline.yml

pipelines:
      default:
        - parallel:
          - step:
              name: Build latest
              image: rust:latest
              script:
                - cargo build --verbose
        - parallel:
          - step:
              name: Build nightly
              image: rustlang/rust:nightly
              script:
                - cargo build --verbose

With just that there is no problems for A when CI run the pipeline, but for B here is next steps:

  1. Go to B repository
  2. Generate an SSH key : Settings -> Pipeline -> SSH Keys -> Generate keys
  3. Update known hosts: Bitbucket Pipelines automatically adds the fingerprint for the Bitbucket and GitHub sites to all pipelines (but doesn't display that in the UI shown above).
  4. Go to the user used to run the pipeline: Settings -> Security -> SSH Keys -> Add key -> Enter a label and copy the public ssh key generated above.
  5. Check that Pipeline is enable for B and A in repository Settings -> Pipelines -> Settings -> Check Enable Pipelines
    And that's it for Bitbucket!
    Full documentation can be found here: Bitbucket Pipelines configuration reference | Bitbucket Cloud | Atlassian Support

GitLab

GitLab need a more complex script than BitBucket because BitBucket do some jobs automatically.
Here is the step to follow if repository B depends of repository A.
The pipeline script for A and B are the not the same :

A .gitlab-ci.yml

# All statges allow to group jobs into stagess
stages:
  - build

# Build with latest Rust 
rust-build-latest:
  stage: build  # This job belong to build stage
  image: rust:latest # Tags of a Docker image
  script: 
    - cargo build --verbose

# Build with nightly Rust 
rust-build-nightly:
  stage: build
  image: rustlang/rust:nightly
  script:
    - cargo build --verbose

Nothings to say here just a simple script that run 2 jobs,1 with latest Rust, 1 with nightly Rust.

B .gitlab-ci.yml

# All statges allow to group jobs into stagess
stages:
  - build
  
# Stuff to do before launching scripts
before_script:
  ##
  ## Install ssh-agent if not already installed, it is required by Docker.
  ## (change apt-get to yum if you use an RPM-based image)
  ##
  - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'

  ##
  ## Run ssh-agent (inside the build environment)
  ##
  - eval $(ssh-agent -s)

  ##
  ## Add the SSH key stored in SSH_PRIVATE_KEY variable to the agent store
  ## We're using tr to fix line endings which makes ed25519 keys work
  ## without extra base64 encoding.
  ## https://gitlab.com/gitlab-examples/ssh-private-key/issues/1#note_48526556
  ##
  - echo SSH_PRIVATE_KEY is "$SSH_PRIVATE_KEY"
  - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -

  ##
  ## Create the SSH directory and give it the right permissions
  ##
  - mkdir -p ~/.ssh
  - chmod 700 ~/.ssh

  ##
  ## Collect SSH RSA public key from gitlab.com and add them to the known_hosts file
  ## 
  - ssh-keyscan -t rsa gitlab.com >> ~/.ssh/known_hosts

  ## 
  ## Test the connection to the gitlab.com hosts
  ## Permanently added the RSA host key for IP address 'X.X.X.X' to the list of known hosts.
  ##
  - ssh -T git@gitlab.com
 
# Build with latest Rust 
rust-build-latest:
  stage: build  # This job belong to build stage
  image: rust:latest # Tags of a Docker image
  script:
    - cargo build --verbose
   
# Build with nightly Rust 
rust-build-nightly:
  stage: build
  image: rustlang/rust:nightly
  script:
    - cargo build --verbose

For GitLab we need to to some steps in the script before launching CI jobs. (This is done automatically with BitBucket)

  • First we manually setup ssh-agent if not already installed and run it inside the build environment.
  • Then we add the content of a variable called SSH_PRIVATE_KEY ( Or other name, this does not matter ) to the ssh agent store.
  • Create and give the right permissions to ~/.ssh directory.
  • Scan the keys of your private server with ssh-keyscan -t rsa gitlab.com >> ~/.ssh/known_hosts
  • Optionally, you can test the connection to the gitlab.com hosts.
    All is here : Using SSH keys with GitLab CI/CD | GitLab
    Concerning the SSH_PRIVATE_KEY, we have to create the key on our computer and go to B repository then Settings -> CI/CD -> Variables -> Create a variable named SSH_PRIVATE_KEY with value containing the ssh private key
    After adding the private key, in the same repository go to Settings -> Repository -> Deploy Keys -> Add the ssh public key here

And that's it for GitLab!

I hope this post will help programmers that spend more time with coding than integrating but want to use CI :wink:

Don't hesitate to give your feedback or comment.

1 Like

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.