Topological insulators (TIs) are exotic materials with strong spin-orbit coupling that are electrically insulating in their interior, but conduct on their surface regardless of surface orientation, material purity, or any other details as long as the interior remains insulating and time-reversal symmetry is preserved. Moreover, the corresponding topological surface states are characterized by spin-momentum locking, which means that the surface currents have a fixed spin polarization correlated with their propagation direction. Furthermore, exotic phenomena such as the quantum anomalous Hall effect, the topological magneto-electric effect, and the appearance of Majorana bound states, were proposed and explored for TIs interfaced with other types of materials. It is therefore interesting to investigate such interfaces and explore how they can be used to construct confined quantum systems with exotic properties.

In this doctoral thesis, we investigated topological states in topological-insulator junctions and confined quantum systems in heterostructures made from TIs interfaced with other materials such as magnetic and superconducting films and graphene, that are deposited on the surface of a topological insulator.

First, we looked at junctions made from two TIs with an additional mirror crystal symmetry whose surface states are spin-momentum locked in opposite directions. In this case, there exist topological crystalline states at the interface between the two TIs. The interface can be probed by measuring the conductance as a function of a rotating parallel magnetic field which periodically breaks and restores the mirror symmetry which leads to conductance oscillations.

Secondly, we considered quantum dots and rings on the surface of a topological insulator, where the surface state is confined with magnetic films or by proximity to a superconductor. We found that electron-electron interactions in magnetically confined quantum dots lead to the formation of spin-polarized Wigner molecules and that hybrid quantum rings consisting of an annulus bounded by magnetic or superconducting regions can support exotic Majorana bound states.

Finally, we studied heterostructures made from a monolayer of graphene that is deposited on the surface of a topological insulator. The interaction with graphene significantly modifies the properties of the topological surface state which migrates to the graphene layer. We discussed different commensurate stacking configurations and derived a low-energy model. Then we considered tunneling of electrons from the bare surface to steps and through nanoribbons of the deposited graphene. We calculated the conductance whose features could be understood from antiresonances in the transmission probability due to quasibound ribbon states.