IPSec, which is short for "IP Security" is the name of a security architecture and set of protocols commonly used to construct a VPN. These services work at the IP (Internet Protocol) or network layer and provide confidentiality and authentication as the packets move through networked devices.
A simplified explanation of the way it works starts with the need for both ends of the conversation to have public/private key pairs. Asymmetric cryptography/PKI is used for each end to authenticate and to negotiate a shared secret key that’s used for the rest of the session. That part of the protocol is called Internet Key Exchange (IKE). Once the secret key is negotiated and shared, protected with public keys, the receiver can be sure that the information wasn’t changed and that it’s from the other party, since only the other party knows that secret key.
IPSec allows one of two modes, one where the packets are encrypted between end points, called transport mode, and one where the packets are encapsulated yet again to go through a gateway device, called tunnel mode.