Update 2022-03-29: It’s worth mentioning that we track this wallet’s issue in our own issue tracker as they don’t have a public issue tracker. It’s issue 340. Also: Today, 6 months later there is no new release for Mk3, so the below verdict of a beningn diff remains our verdict.

Update 2021-09-07: The provider answered our mails. Here is the full correspondence:

Full Correspondence

As the provider doesn’t maintain a public issue tracker, it’s a bit hard to be fully transparent but we try our best …

Leo Wandersleb 2021-08-13:

Hi

as shared on Twitter https://twitter.com/WalletScrutiny/status/1426227415563526155 and on WalletScrutiny https://walletscrutiny.com/hardware/coldcardMk3/ I had issues reproducing the latest build. Namely the verification script doesn’t find the downloaded binary in the releases folder.

The closest to what your GitHub instructs + what I understand should be done can be found in the attached log file.

Please advise where I went wrong.

Kind Regards,

Leo Wandersleb

support@coinkite.com 2021-08-13:

Hi Leo,

You’ll need to fetch the binary you are reproducing from here:

https://coldcardwallet.com/downloads/

Put that into the releases subdir of the project… but that needs to happen inside the Docker image. I don’t remember the mechanics of that off the top of my head.

I will raise it with the technical team. We probably need to update our docs to make this process easier to follow.

Some feedback on: https://walletscrutiny.com/hardware/coldcardMk3/

• the secure element we use is the ATECC608B and it is used to store the seed phrase and not bitcoin-releated crypto (same system architecture BitBox has adopted)
• the PIN-retry limiting is implemented inside the secure element, and no means is provided to “reset” that counter (by design) so we can’t recover those units with unknown PIN codes. After 3 wrong tries, and up to the final 13th, we give lots of warnings and hints (ie. we show it on-screen) before each attempt.
• entropy from the TRNG in the main micro is used if the seed is generated by the Coldcard (it is also whitened by SHA256); TRNG in secure element is not used except to resist replay attacks when communicating with it.
• you can also enter D6 dice rolls, and generate a seed phrase that way (using the Coldcard) we show how to verify the process is correct, on this page: https://coldcardwallet.com/docs/verifying-dice-roll-math
• I would suggest you inspect and score hardware wallets for which crypto libraries they use; Coldcard recently upgraded to libsecp256k1 written by the Bitcoin Core devs, which appears to be best-in-class
• hardware wallets can effectively leak your private key by using known, or weak K values inside signatures. This makes every transaction an opportunity to leak your private key directly onto the blockchain… Coldcard uses deterministic K values per RFC 6979, and it can be verified that we do (both in the source, our test vectors, and in fielded devices)

Looking forward to updates, please keep us in the loop.

-Coinkite Team

Leo Wandersleb 2021-08-13:

Hi Team,

Hi Leo,

You’ll need to fetch the binary you are reproducing from here:

https://coldcardwallet.com/downloads/ https://coldcardwallet.com/downloads/

I did that. The log should show it.

Put that into the releases subdir of the project…

I did that. Please verify in the log.

but that needs to happen inside the Docker image. I don’t remember the mechanics of that off the top of my head.

Not sure what you mean by “inside the docker image” but inside the docker image the file is available. If I run the image interactively with the same volumes mounted (I think), I see it:

$ docker run --rm -it --volume=$(realpath ..):/work/src:ro --volume=$(realpath built):/work/built:rw --privileged coldcard-build bash
bash-5.1# ls -l /work/src/releases/2021-07-28T1347-v4.1.2-coldcard.dfu
-rw-rw-r--    1 1000     1000        757053 Aug 13 18:48 /work/src/releases/2021-07-28T1347-v4.1.2-coldcard.dfu

I will raise it with the technical team. We probably need to update our docs to make this process easier to follow.

Some feedback on: https://walletscrutiny.com/hardware/coldcardMk3/ https://walletscrutiny.com/hardware/coldcardMk3/

Cool! Thanks for the feedback!

• the secure element we use is the ATECC608B and it is used to store the seed phrase and not bitcoin-releated crypto (same system architecture BitBox has adopted)

… which makes a lot of sense. Where can I find this in a “public statement of the provider” to reference in my analysis?

• the PIN-retry limiting is implemented inside the secure element, and no means is provided to “reset” that counter (by design) so we can’t recover those units with unknown PIN codes. After 3 wrong tries, and up to the final 13th, we give lots of warnings and hints (ie. we show it on-screen) before each attempt.

I understood that part I think. I did not understand why a factory reset would not be the most logical thing instead of a bricking. Without a clear security advantage, this feels like it’s designed to make customers buy more devices cause forgetting pins is such a common thing to do.

• entropy from the TRNG in the main micro is used if the seed is generated by the Coldcard (it is also whitened by SHA256); TRNG in secure element is not used except to resist replay attacks when communicating with it.

Do you have a quotable source? Is the entropy verifiable?

• you can also enter D6 dice rolls, and generate a seed phrase that way (using the Coldcard) we show how to verify the process is correct, on this page: https://coldcardwallet.com/docs/verifying-dice-roll-math

https://coldcardwallet.com/docs/verifying-dice-roll-math

I knew that but maintain WalletScrutiny with default behavior in mind. If the user is by default asked to get a dice and provide at least 10 numbers, I’ll happily add that.

• I would suggest you inspect and score hardware wallets for which crypto libraries they use; Coldcard recently upgraded to libsecp256k1 written by the Bitcoin Core devs, which appears to be best-in-class

I have a lot of plans to improve the security reporting. So far it’s exclusively about transparency. As anything but blatant backdoors is open to interpretation and fiercely debated between hardware wallet providers, I was thinking of “expert opinions” and wallet developers would all qualify to opine on the security of competing wallets.

For example I highly regard ColdCard being BTC only as it massively reduces the attack surface compared to others that put libraries from 100 teams on their hardware.

• hardware wallets can effectively leak your private key by using known, or weak K values inside signatures. This makes every transaction an opportunity to leak your private key directly onto the blockchain… Coldcard uses deterministic K values per RFC 6979, and it can be verified that we do (both in the source, our test vectors, and in fielded devices)

I know (and tweeted about this specific way of leaking before) but got challenged to call out any modern wallet that hasn’t adopted deterministic signatures. I worked for Mycelium until two months ago and they fixed this before I joined five years ago. Please help me call out any wallet that still doesn’t sign deterministically!

support@coinkite.com 2021-08-16:

Hi Leo,

Not sure what you mean by “inside the docker image” but inside the docker image the file is available. If I run the image interactively with the same volumes mounted (I think), I see it:

$ docker run --rm -it --volume=$(realpath ..):/work/src:ro --volume=$(realpath built):/work/built:rw --privileged coldcard-build bash
bash-5.1# ls -l /work/src/releases/2021-07-28T1347-v4.1.2-coldcard.dfu
-rw-rw-r--    1 1000     1000        757053 Aug 13 18:48 /work/src/releases/2021-07-28T1347-v4.1.2-coldcard.dfu

That should work but I’m not a docker expert. It would be great if you could make a pull-request once you’ve got it smooth again. My team will also look at it (and the docs) and see if we can fix it too.

• the secure element we use is the ATECC608B and it is used to store the seed phrase and not bitcoin-releated crypto (same system architecture BitBox has adopted)

… which makes a lot of sense. Where can I find this in a “public statement of the provider” to reference in my analysis?

https://coldcardwallet.com/docs/faq under heading “Is the secure element’s crypto used for Bitcoin processing?” ​

I knew that but maintain WalletScrutiny with default behavior in mind. If the user is by default asked to get a dice and provide at least 10 numbers, I’ll happily add that.

We do support that as well but don’t require it from our users. After Coldcard has picked a seed, the user can add some more dice rolls into the entropy. That’s a different mode and not what I was talking about… 10 rolls would only be ~20 bits more entropy, and so IMHO not too useful. (BTW, we sell a package of 100 dice so full 256-entropy can be rolled in one shot.)

deterministic K values per RFC 6979, and it can be verified that we do (both in the source, our test vectors, and in fielded devices)

I know (and tweeted about this specific way of leaking before) but got challenged to call out any modern wallet that hasn’t adopted deterministic signatures. I worked for Mycelium until two months ago and they fixed this before I joined five years ago. Please help me call out any wallet that still doesn’t sign deterministically!

If an evil wallet was to do this, it would not be obvious from the source code, and their code would build deterministically. An outside test lab, such as yourselves, should be building every release of every wallet and checking that signatures remain identical to each other. In fact, all wallets should be producing the exactly same signature bytes if they are using the same private key. I think, given the same PSBT file (via HWI) they should all produce the same signed transaction (but might be some edge cases I’m forgetting).

-Coinkite Team

support@coinkite.com 2021-09-01:

Hi Leo,

Some fixes have been made to the repro. Please pull the latest.

It will show this at the end:

diff repro-got.txt repro-want.txt
--- repro-got.txt
+++ repro-want.txt
@@ -41390,7 +41390,7 @@
 000a22e0  65 78 69 74 0d 0a 00 52  01 00 4d 69 63 72 6f 50  |exit...R..MicroP|
 000a22f0  79 74 68 6f 6e 20 76 31  2e 39 2e 33 2d 33 36 30  |ython v1.9.3-360|
 000a2300  32 2d 67 35 39 31 37 66  63 31 39 39 2d 64 69 72  |2-g5917fc199-dir|
-000a2310  74 79 20 6f 6e 20 32 30  32 31 2d 30 39 2d 30 31  |ty on 2021-09-01|
+000a2310  74 79 20 6f 6e 20 32 30  32 31 2d 30 37 2d 32 38  |ty on 2021-07-28|
 000a2320  3b 20 43 6f 6c 64 63 61  72 64 20 77 69 74 68 20  |; Coldcard with |
 000a2330  53 54 4d 33 32 4c 34 78  78 52 47 0d 0a 00 54 79  |STM32L4xxRG...Ty|
 000a2340  70 65 20 22 68 65 6c 70  28 29 22 20 66 6f 72 20  |pe "help()" for |
@@ -42360,7 +42360,7 @@
 000a5fd0  68 20 53 54 4d 33 32 4c  34 78 78 52 47 00 76 31  |h STM32L4xxRG.v1|
 000a5fe0  2e 39 2e 33 2d 33 36 30  32 2d 67 35 39 31 37 66  |.9.3-3602-g5917f|
 000a5ff0  63 31 39 39 2d 64 69 72  74 79 20 6f 6e 20 32 30  |c199-dirty on 20|
-000a6000  32 31 2d 30 39 2d 30 31  00 31 2e 31 33 2e 30 00  |21-09-01.1.13.0.|
+000a6000  32 31 2d 30 37 2d 32 38  00 31 2e 31 33 2e 30 00  |21-07-28.1.13.0.|
 000a6010  70 79 62 6f 61 72 64 00  c0 5c 05 08 01 d4 03 08  |pyboard..\......|
 000a6020  78 63 05 08 28 e0 0a 08  44 54 05 08 9f 00 00 00  |xc..(...DT......|
 000a6030  13 00 00 00 38 e0 0a 08  ba 00 00 00 42 2d 00 00  |....8.......B-..|
make: *** [Makefile:279: check-repro] Error 1

​ If you look closely, it’s just a build timestamp that got into the binary. We will fix this in the next release. As you can see from the diff, all executable bytes are unchanged.

-Coinkite Team

support@coinkite.com 2021-09-01:

Hi Leo,

Also a new firmware version will come out today or tomorrow, so you might want to wait for that.

-Coinkite Team

The Analysis

The provider makes clear claims:

COLDCARD Hardware Wallet
✓ Bitcoin Only
✓ Open-Source
✓ Easy-to-Use
✓ Ultra-Secure
✓ Loved by Cypherpunks

and on their repository: Reproducible Builds:

To have confidence this source code tree is the same as the binary on your device, you can rebuild it from source and get exactly the same bytes.

Also the product is packed with security features, some unique to the Coldcard Mk3.

An anti-feature we would like to know why it’s actually a feature is

Because of the in-depth use of the secure element, there is no “factory reset” for the Coldcard. If you forget your Coldcard PIN, there is nothing we can do except remind you to recycle your e-waste responsibly!

meaning that if you ever forget your PIN even if you have your seed phrase, your device is worthless. To our knowledge, no other device works like this and we see no good reason why this could be advantageous to the user or more secure.

The “secure element” comes with another implication: Code run on the “secure element” usually cannot be audited. “Secure element” providers require NDAs and closed source from the users. As we’ve seen with BitBox02 Unreproducible! Review Outdated! , this doesn’t need to be to the detriment of the product’s security: If the security doesn’t rely on the “secure element” but only uses it supplementary, the product can benefit from added security without putting funds at risk if the “secure element” is evil.

It turns out a bit hard to find what aspects Coldcard Mk3 delegates to the “secure element”. Is there a single point of failure? Is the masterseed by default generated with entropy solely from the “secure element”? If so, the closed source secure element might be generating backups from poor randomness that are trivial to recreate by the manufacturer of the chip. This would put the product as a whole into our “closed source” category!

Surprisingly there is no public issue tracker on their firmware repository neither. Let’s see if we get a reply to our question on Twitter.

Reproducing The Firmware

So the complete section on Reproducible Builds is:

To have confidence this source code tree is the same as the binary on your device, you can rebuild it from source and get exactly the same bytes. This process has been automated using Docker. Steps are as follows:

1. Install Docker and start it.
2. Install make (GNUMake) if you don’t already have it.
3. Checkout the code, and start the process.

   git clone https://github.com/Coldcard/firmware.git
   cd firmware/stm32
   make repro
   

4. At the end of the process a clear confirmation message is shown, or the differences.
5. Build products can be found firmware/stm32/built.

Let’s see … we have docker installed and running aka “started” and make is available, too. Let’s see what make repro would run on our machine first:

$ git clone https://github.com/Coldcard/firmware.git; cd firmware/stm32
$ cat Makefile 
# (c) Copyright 2018 by Coinkite Inc. This file is covered by license found in COPYING-CC.
#
# Build micropython for stm32 (an ARM processor). Also handles signing of resulting firmware images.
#
MPY_TOP = ../external/micropython
PORT_TOP = $(MPY_TOP)/ports/stm32
MPY_CROSS = $(MPY_TOP)/mpy-cross/mpy-cross
PYTHON_MAKE_DFU = $(MPY_TOP)/tools/dfu.py
PYTHON_DO_DFU = $(MPY_TOP)/tools/pydfu.py

# aka ../cli/signit.py
SIGNIT = signit

MAKE_ARGS = BOARD=COLDCARD -j 4 EXCLUDE_NGU_TESTS=1

all: COLDCARD/file_time.c
	cd $(PORT_TOP) && $(MAKE) $(MAKE_ARGS)

clean:
	cd $(PORT_TOP) && $(MAKE) $(MAKE_ARGS) clean
	git clean -xf built

# These trigger the 'all' target when we haven't completed a successful build yet
l-port/build-COLDCARD/firmware.elf: all
l-port/build-COLDCARD/firmware0.bin: all
l-port/build-COLDCARD/firmware1.bin: all

firmware.elf: l-port/build-COLDCARD/firmware.elf
	cp l-port/build-COLDCARD/firmware.elf .

# These values used to make .DFU files. Flash memory locations.
FIRMWARE_BASE   = 0x08008000
BOOTLOADER_BASE = 0x08000000
FILESYSTEM_BASE = 0x080e0000

# Our version for this release.
VERSION_STRING = 4.1.2

#
# Sign and merge various parts
#
firmware-signed.bin: l-port/build-COLDCARD/firmware0.bin l-port/build-COLDCARD/firmware1.bin
	$(SIGNIT) sign $(VERSION_STRING) -o $@
firmware-signed.dfu: firmware-signed.bin Makefile
	$(PYTHON_MAKE_DFU) -b $(FIRMWARE_BASE):$< $@

# make the DFU file which is shared for upgrades
dfu: firmware-signed.dfu

# Build a binary, signed w/ production key
# - always rebuild binary for this one
.PHONY: dev.dfu
dev.dfu: l-port/build-COLDCARD/firmware?.bin
	cd $(PORT_TOP) && $(MAKE) $(MAKE_ARGS)
	$(SIGNIT) sign $(VERSION_STRING) -k 1 -o dev.bin
	$(PYTHON_MAKE_DFU) -b $(FIRMWARE_BASE):dev.bin dev.dfu

.PHONY: remake
remake:
	rm -rf l-port/build-COLDCARD/firmware?.bin l-port/build-COLDCARD/frozen_mpy*

# This is fast for Coinkite devs, but no DFU support in the wild.
up: dev.dfu
	$(PYTHON_DO_DFU) -u dev.dfu

# Slow, but works with unmod-ed board: use USB protocol to upgrade (2 minutes)
dev: dev.dfu
	ckcc upgrade dev.dfu

COLDCARD/file_time.c: Makefile make_filetime.py
	./make_filetime.py COLDCARD/file_time.c $(VERSION_STRING)

# Make a factory release: using key #1
# - when executed in a repro w/o the required key, it defaults to key zero
# - and that's what happens inside the Docker build
production.bin: firmware-signed.bin Makefile
	$(SIGNIT) sign $(VERSION_STRING) -r firmware-signed.bin -k 1 -o $@

# This is release of the bootloader that will be built into the release firmware.
BOOTLOADER_VERSION = 2.0.1

.PHONY: release
release: code-committed
	$(MAKE) clean
	$(MAKE) repro
	test -f built/production.bin
	$(MAKE) release-products
	$(MAKE) tag-source

# Make a release-candidate, faster.
.PHONY: rc1
rc1: 
	$(MAKE) repro
	test -f built/production.bin
	$(MAKE) release-products

# This target just combines latest version of production firmware with bootrom into a DFU
# file, stored in ../releases with appropriately dated file name.
.PHONY: release-products
release-products: NEW_VERSION = $(shell $(SIGNIT) version built/production.bin)
release-products: RELEASE_FNAME = ../releases/$(NEW_VERSION)-coldcard.dfu
release-products: built/production.bin
	test ! -f $(RELEASE_FNAME)
	cp built/file_time.c COLDCARD/file_time.c
	-git commit COLDCARD/file_time.c -m "For $(NEW_VERSION)"
	$(SIGNIT) sign $(VERSION_STRING) -r built/production.bin -k 1 -o built/production.bin
	$(PYTHON_MAKE_DFU) -b $(FIRMWARE_BASE):built/production.bin \
		-b $(BOOTLOADER_BASE):bootloader/releases/$(BOOTLOADER_VERSION)/bootloader.bin \
		$(RELEASE_FNAME)
	@echo
	@echo 'Made release: ' $(RELEASE_FNAME)
	@echo

built/production.bin:
	@echo "To make production build, must run docker code"
	@false

# Use DFU to install the latest production version you have on hand
dfu-latest: 
	$(PYTHON_DO_DFU) -u `ls -t1 ../releases/*.dfu | head -1`

# Use slow USB upload and reboot method.
latest:
	ckcc upgrade `ls -t1 ../releases/*.dfu | head -1`

.PHONY: code-committed
code-committed:
	@echo ""
	@echo "Are all changes commited already?"
	git diff --stat --exit-code .
	@echo '... yes'

# Sign a message with the contents of ../releases on the developer's machine
.PHONY: sign-release
sign-release:
	(cd ../releases; shasum -a 256 *.dfu *.md | sort -rk 2 | \
		gpg --clearsign -u A3A31BAD5A2A5B10 --digest-algo SHA256 --output signatures.txt --yes - )
	git commit -m "Signed for release." ../releases/signatures.txt

# Tag source code associate with built release version.
# - do "make release" before this step!
# - also edit/commit ChangeLog text too
# - update & sign signatures file
# - and tag everything
tag-source: PUBLIC_VERSION = $(shell $(SIGNIT) version built/production.bin)
tag-source: sign-release code-committed
	git commit  --allow-empty -am "New release: "$(PUBLIC_VERSION)
	echo "Taging version: " $(PUBLIC_VERSION)
	git tag -a $(PUBLIC_VERSION) -m "Release "$(PUBLIC_VERSION)
	git push
	git push --tags

# DFU file of boot and main code
# - bootloader is last so it can fail if already installed (maybe)
#
mostly.dfu: firmware-signed.bin bootloader/bootloader.bin Makefile
	$(PYTHON_MAKE_DFU) \
			-b $(FIRMWARE_BASE):firmware-signed.bin \
			-b $(BOOTLOADER_BASE):bootloader/bootloader.bin $@

# send everything
m-dfu: mostly.dfu
	$(PYTHON_DO_DFU) -u mostly.dfu

# Clear the internal filesystem (for dev-mistakes recovery)
# - unused?
.PHONY: wipe-fs
wipe-fs: 
	dd if=/dev/urandom of=tmp.bin bs=512 count=1
	$(PYTHON_MAKE_DFU) -b $(FILESYSTEM_BASE):tmp.bin tmp.dfu
	$(PYTHON_DO_DFU) -u tmp.dfu
	rm tmp.bin tmp.dfu

# unused
stlink:
	cd $(PORT_TOP) && $(MAKE) $(MAKE_ARGS) deploy-stlink

# useless, will be ignored by bootloader
unsigned-dfu:
	cd $(PORT_TOP) && $(MAKE) $(MAKE_ARGS) deploy

# see COLDCARD/mpconfigboard.mk
tags: 
	cd $(PORT_TOP) && $(MAKE) $(MAKE_ARGS) tags
checksum: 
	cd $(PORT_TOP) && $(MAKE) $(MAKE_ARGS) checksum
files:
	cd $(PORT_TOP) && $(MAKE) $(MAKE_ARGS) files

# OLD dev junk?
# compile and freeze python code
PY_FILES = $(shell find ../shared -name \*.py)
ALL_MPY_FILES = $(addprefix build/, $(PY_FILES:../shared/%.py=%.mpy))
MPY_FILES = $(filter-out build/obsolete/%, $(ALL_MPY_FILES))

# In another window: 
#
#		openocd -f openocd_stm32l4x6.cfg
#
# Can do:
# - "load" which writes the flash (medium speed, lots of output on st-util)
# - "cont" starts/continues system
# - "br main" sets breakpoints
# - "mon reset" to reset micro
# - and so on
#
debug:
	arm-none-eabi-gdb built/firmware.elf -x gogo.gdb

# detailed listing, very handy
OBJDUMP = arm-none-eabi-objdump
firmware.lss: l-port/build-COLDCARD/firmware.elf
	$(OBJDUMP) -h -S $< > $@

# Dump sizes of all frozen py files; requires recent build.
.PHONY: sizes
sizes:
	wc -c l-port/build-COLDCARD/frozen_mpy/*.mpy | sort -n

# Measure flash impact of a single file. Great for before/after.
# 	make F=foo.py size
# where: foo.py is anything in ../shared
size:
	$(MPY_CROSS) -o tmp.mpy -s $F ../shared/$F
	wc -c tmp.mpy

# one time setup, after repo checkout
setup:
	cd $(MPY_TOP) ; git submodule update --init lib/stm32lib
	cd $(MPY_TOP)/lib/stm32lib ; sed -i.orig -e 's/#define VECT_TAB_OFFSET  0x00/    /' \
				CMSIS/STM32L4xx/Source/Templates/system_stm32l4xx.c
	cd ../external/libngu; make min-one-time
	cd $(MPY_TOP)/mpy-cross ; make
	-ln -s $(PORT_TOP) l-port
	-ln -s $(MPY_TOP) l-mpy
	cd $(PORT_TOP)/boards; if [ ! -L COLDCARD ]; then \
		ln -s ../../../../../stm32/COLDCARD COLDCARD; fi
	

# Caution: docker container has read access to your source tree
# - a readonly copy of source tree, and one output directory
# - build products are copied to there, see repro-build.sh
# - works from this repo, but starts with copy of HEAD
DOCK_RUN_ARGS = -v $(realpath ..):/work/src:ro \
				-v $(realpath built):/work/built:rw \
				--privileged coldcard-build
repro: code-committed
	docker build -t coldcard-build - < dockerfile.build
	(cd ..; docker run $(DOCK_RUN_ARGS) sh src/stm32/repro-build.sh)

# debug: shell into docker container
shell:
	docker run -it $(DOCK_RUN_ARGS) sh

# debug: allow docker to write into source tree
#DOCK_RUN_ARGS := -v $(realpath ..):/work/src:rw --privileged coldcard-build

PUBLISHED_BIN = $(wildcard ../releases/*-v$(VERSION_STRING)-coldcard.dfu)

# final step in repro-building: check you got the right bytes
# - but you don't have the production signing key, so that section is removed
check-repro: TRIM_SIG = sed -e 's/^00003f[89abcdef]0 .*/(firmware signature here)/'
check-repro: firmware-signed.bin
ifeq ($(PUBLISHED_BIN),)
	@echo ""
	@echo "Need published binary for: $(VERSION_STRING)"
	@echo ""
	@echo "Copy it into ../releases"
	@echo ""
else
	@echo Comparing against: $(PUBLISHED_BIN)
	test -n "$(PUBLISHED_BIN)" -a -f $(PUBLISHED_BIN)
	$(RM) -f check-fw.bin check-bootrom.bin
	$(SIGNIT) split $(PUBLISHED_BIN) check-fw.bin check-bootrom.bin
	$(SIGNIT) check check-fw.bin
	$(SIGNIT) check firmware-signed.bin
	hexdump -C firmware-signed.bin | $(TRIM_SIG) > repro-got.txt
	hexdump -C check-fw.bin | $(TRIM_SIG) > repro-want.txt
	diff repro-got.txt repro-want.txt
	@echo ""
	@echo "SUCCESS. "
	@echo ""
	@echo "You have built a bit-for-bit identical copy of Coldcard firmware for v$(VERSION_STRING)"
endif


# EOF

which is a scary lot of stuff we are supposed to run on our machine instead of in a sandbox but it all boils down to those few lines:

DOCK_RUN_ARGS = -v $(realpath ..):/work/src:ro \
				-v $(realpath built):/work/built:rw \
				--privileged coldcard-build
	docker build -t coldcard-build - < dockerfile.build
	(cd ..; docker run $(DOCK_RUN_ARGS) sh src/stm32/repro-build.sh)

Let’s see …

$ docker build --tag coldcard-build - < dockerfile.build
$ DOCK_RUN_ARGS="--volume=$(realpath ..):/work/src:ro --volume=$(realpath built):/work/built:rw --privileged coldcard-build"
$ (cd ..; docker run --rm $DOCK_RUN_ARGS sh src/stm32/repro-build.sh)
...
Need published binary for: 4.1.2

Copy it into ../releases

Ok, that did something and as the latest commit looks like it’s tagged with the latest release v4.1.2:

$ git log -n 1
commit 4f69140ded9168b6a372a37540554e9099e065af (HEAD -> master, tag: 2021-07-28T1347-v4.1.2, origin/master, origin/HEAD)
Author: Peter D. Gray <peter@conalgo.com>
Date:   Wed Jul 28 09:47:37 2021 -0400

    New release: 2021-07-28T1347-v4.1.2

we are supposed to re-run this with the published binary in ../releases … relative to where? There is a ../releases aka repositoryFolder/releases but there we get the same issue:

$ wget https://coldcardwallet.com/downloads/2021-07-28T1347-v4.1.2-coldcard.dfu
$ sha256sum 2021-07-28T1347-v4.1.2-coldcard.dfu 
d01d81305b209dadcf960b9e9d20affb8d4f11e9f9f916c5a06be29298c80dc2  2021-07-28T1347-v4.1.2-coldcard.dfu
$ mv 2021-07-28T1347-v4.1.2-coldcard.dfu ../releases/
$ (cd ..; docker run --rm $DOCK_RUN_ARGS sh src/stm32/repro-build.sh)
...
Need published binary for: 4.1.2

Copy it into ../releases

So just to be safe this is not an issue with the shortcut we took by not running make repro and neither an issue with ../releases/ referring to a folder further up in the hierarchy, we tried with make repro, with 2021-07-28T1347-v4.1.2-coldcard.dfu in both the existing firmware/releases and a new firmware/../releases to no avail. For now, this firmware is not verifiable.

Update 2021-09-08

With the latest correspondence and updated repo, we give it another try:

$ git clone https://github.com/Coldcard/firmware.git
$ cd firmware/stm32
$ make repro
...
signit split ../releases/2021-09-02T1752-v4.1.3-coldcard.dfu check-fw.bin check-bootrom.bin
start 293 for 727040 bytes: Firmware => check-fw.bin
start 727341 for 30720 bytes: Bootrom => check-bootrom.bin
signit check check-fw.bin
     magic_value: 0xcc001234
       timestamp: 2021-09-02 17:52:57 UTC
  version_string: 4.1.3
      pubkey_num: 1
 firmware_length: 727040
   install_flags: 0x0 =>
       hw_compat: 0x6 => Mk2+Mk3
          future: 0000000000000000 ... 0000000000000000
       signature: f3ae1cab1232c5e3 ... 242dd1f30ff498b7
 ECDSA Signature: CORRECT
signit check firmware-signed.bin
     magic_value: 0xcc001234
       timestamp: 2021-09-08 00:22:48 UTC
  version_string: 4.1.3
      pubkey_num: 0
 firmware_length: 727040
   install_flags: 0x0 =>
       hw_compat: 0x6 => Mk2+Mk3
          future: 0000000000000000 ... 0000000000000000
       signature: 4b333eccfa13cfca ... ab4245b0e0be6dea
 ECDSA Signature: CORRECT
hexdump -C firmware-signed.bin | sed -e 's/^00003f[89abcdef]0 .*/(firmware signature here)/' > repro-got.txt
hexdump -C check-fw.bin | sed -e 's/^00003f[89abcdef]0 .*/(firmware signature here)/' > repro-want.txt
diff repro-got.txt repro-want.txt
--- repro-got.txt
+++ repro-want.txt
@@ -41444,7 +41444,7 @@
 000a2640  0d 0a 00 52 01 00 4d 69  63 72 6f 50 79 74 68 6f  |...R..MicroPytho|
 000a2650  6e 20 76 31 2e 39 2e 33  2d 33 36 30 32 2d 67 35  |n v1.9.3-3602-g5|
 000a2660  39 31 37 66 63 31 39 39  2d 64 69 72 74 79 20 6f  |917fc199-dirty o|
-000a2670  6e 20 32 30 32 31 2d 30  39 2d 30 38 3b 20 43 6f  |n 2021-09-08; Co|
+000a2670  6e 20 32 30 32 31 2d 30  39 2d 30 32 3b 20 43 6f  |n 2021-09-02; Co|
 000a2680  6c 64 63 61 72 64 20 77  69 74 68 20 53 54 4d 33  |ldcard with STM3|
 000a2690  32 4c 34 78 78 52 47 0d  0a 00 54 79 70 65 20 22  |2L4xxRG...Type "|
 000a26a0  68 65 6c 70 28 29 22 20  66 6f 72 20 6d 6f 72 65  |help()" for more|
@@ -42415,7 +42415,7 @@
 000a6330  4d 33 32 4c 34 78 78 52  47 00 76 31 2e 39 2e 33  |M32L4xxRG.v1.9.3|
 000a6340  2d 33 36 30 32 2d 67 35  39 31 37 66 63 31 39 39  |-3602-g5917fc199|
 000a6350  2d 64 69 72 74 79 20 6f  6e 20 32 30 32 31 2d 30  |-dirty on 2021-0|
-000a6360  39 2d 30 38 00 31 2e 31  33 2e 30 00 70 79 62 6f  |9-08.1.13.0.pybo|
+000a6360  39 2d 30 32 00 31 2e 31  33 2e 30 00 70 79 62 6f  |9-02.1.13.0.pybo|
 000a6370  61 72 64 00 c0 5c 05 08  01 d4 03 08 78 63 05 08  |ard..\......xc..|
 000a6380  84 e3 0a 08 44 54 05 08  9f 00 00 00 13 00 00 00  |....DT..........|
 000a6390  94 e3 0a 08 ba 00 00 00  42 2d 00 00 fa 2c 00 00  |........B-...,..|
make: *** [Makefile:279: check-repro] Error 1
+ set +ex

This is indeed about what they claimed we should have gotten for the v4.1.2 release. So we file v4.1.3 (this release) as not reproducible although the diff looks harmless but would still like to know how we can check the v4.1.2 release as we got no diff at all.

There is more issues with the here presented way of checking a binary: The above script did not check a binary. It did download a binary which it checked against the local build. This might be a different binary than the one we wanted to check.

And as explained elsewhere, the analysis if the test we just did actually is a valid test is considered the responsibility of a proper code review. The code that runs the test is part of the firmware’s repository and our claim is that if the code is clean then so is the binary, in this case, except for a date string.

(ml, lw)

Verdict Explained

We could not verify that the provided code matches the binary!

As part of our Methodology, we ask:

But we also ask: