Dilawar's Blog

watch -n 1 /dev/null | moved to https://dilawars.notion.site

Fixing git clone over https - Server certificate verification failed

  Dilawar Singh

As soon as I try to clone using git clone https://gitlab.subcom.tech/subcom/voicedb, I run into the following problem:

admin@ip-172-26-13-71:~/Work$ git clone https://gitlab.subcom.tech/subcom/voicedb.git
Cloning into 'voicedb'...
fatal: unable to access 'https://gitlab.subcom.tech/subcom/voicedb.git/': server certificate verification failed. CAfile: none CRLfile: none

This is a big problem. The only protocol which works out of box is ssh. To use ssh protocol, everyone has to upload public key to the server. A bit of hassle if you are not working on your own machine :sad:.

First thing first, there is an issue with certificate. Lets debug this.

Check for certificates

First, I’d like to know how certificates behaves on a working site.

Base case: github.com certificates.

admin@ip-172-26-7-193:~$ openssl s_client -showcerts -servername github.com -connect github.com:443 </dev/null 2>/dev/null
CONNECTED(00000003)
---
Certificate chain
 0 s:C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = github.com
   i:C = US, O = "DigiCert, Inc.", CN = DigiCert High Assurance TLS Hybrid ECC SHA256 2020 CA1
-----BEGIN CERTIFICATE-----
... redacted ...
-----END CERTIFICATE-----
 1 s:C = US, O = "DigiCert, Inc.", CN = DigiCert High Assurance TLS Hybrid ECC SHA256 2020 CA1
   i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV Root CA
-----BEGIN CERTIFICATE-----
... redacted ...
-----END CERTIFICATE-----
---
Server certificate
subject=C = US, ST = California, L = San Francisco, O = "GitHub, Inc.", CN = github.com

issuer=C = US, O = "DigiCert, Inc.", CN = DigiCert High Assurance TLS Hybrid ECC SHA256 2020 CA1

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2708 bytes and written 366 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

Everything smoothly and now I know the expected output. I don’t understand all of it but I can make some sense out of it. Now let’s try on my gitlab instance.

gitlab.subcom.tech certificates

admin@ip-172-26-7-193:~$ openssl s_client -showcerts -servername gitlab.subcom.tech -connect gitlab.subcom.tech:443 </dev/null 2>/dev/null
CONNECTED(00000003)
---
Certificate chain
 0 s:CN = gitlab.subcom.tech
   i:C = US, O = Let's Encrypt, CN = R3
-----BEGIN CERTIFICATE-----
... redacted ...
-----END CERTIFICATE-----
---
Server certificate
subject=CN = gitlab.subcom.tech

issuer=C = US, O = Let's Encrypt, CN = R3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1884 bytes and written 390 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 21 (unable to verify the first certificate)
---

All right, the problem seems to be that we are unable to verify the first certificate.. Seems like there are multiple of certificates involved; and we are unable to verify the first certificate itself!

To add to the confusion, the site works fine in browser https://gitlab.subcom.tech. This means that certificates (I downloaded them using certbot) are at least fine for browser. The git clone behaviour is definately different than the browser.

I did an internet sortie. Some useful pointers: https://docs.microsoft.com/en-us/archive/blogs/phkelley/adding-a-corporate-or-self-signed-certificate-authority-to-git-exes-store offers a solution. But still very confused.

I search in gitlab documentation. Their documentation is good and I got a helpful hit: https://docs.gitlab.com/omnibus/settings/ssl.html#common-ssl-errors. I had to google few terms and read a bit more. For a person like me who has short attention span, this is the hardest part. But after spending enough hours, I finally figured out the solution.

Solution

admin@ip-172-26-7-193:/etc/gitlab/ssl$ ls /etc/gitlab/ssl/ -l
total 4
lrwxrwxrwx 1 root root   54 Apr  4 19:21 gitlab.subcom.tech.crt -> /etc/letsencrypt/live/gitlab.subcom.tech/cert.pem
lrwxrwxrwx 1 root root   52 Mar 12 15:25 gitlab.subcom.tech.key -> /etc/letsencrypt/live/gitlab.subcom.tech/privkey.pem
-r-------- 1 root root 1675 Mar 15 15:40 gitlab.subcom.tech.key-staging

Note that gitlab.subcom.tech.crt and gitlab.subcom.tech.key are links to certificate and keys downloaded using certbot. The problem here is with cert.pem which does not contain the “chain” of certificates but rather a single certificate. This worked in browser but not using git (curious!).

After I realized that the chain of certificate is missing, I changed the gitlab.subcom.tech.crt link.

admin@ip-172-26-7-193:/etc/gitlab/ssl$ ls /etc/gitlab/ssl/ -l
total 4
lrwxrwxrwx 1 root root   54 Apr  4 19:21 gitlab.subcom.tech.crt -> /etc/letsencrypt/live/gitlab.subcom.tech/fullchain.pem
lrwxrwxrwx 1 root root   52 Mar 12 15:25 gitlab.subcom.tech.key -> /etc/letsencrypt/live/gitlab.subcom.tech/privkey.pem
-r-------- 1 root root 1675 Mar 15 15:40 gitlab.subcom.tech.key-staging

Now the file (rather a link) /etc/gitlab/ssl/gitlab.subcom.tech.crt points to /etc/letsencrypt/live/gitlab.subcom.tech/fullchain.pem rather than to /etc/letsencrypt/live/gitlab.subcom.tech/cert.pem. This solved the issue for me. fullchain.pem has all the certificates needed for verification to be successful.

Finally, I did a gitlab-ctl reconfigure followed by gitlab.ctl restart. I can now clone the repositories over https protocol from ‘gitlab.subcom.tech’ server. Phew!