Forget about performance, there are way more interesting differences between the two languages (usually to the advantage of Rust).
For instance, generics are more powerful in Rust; Swift has trouble expressing some very fundamental kinds of constraints, and const evaluation is nowhere to be found in Swift.
Another important thing that Swift doesn't provide is static thread safety. Although there are ergonomic concurrency APIs (libdispatch), they can still be misused, as thread safety is not part of the type system in Swift.
There's also the question of compile-time metaprogramming and extensibility. Rust provides macros (declarative as well as procedural), while Swift doesn't have such user-facing extensibility. Some built-in protocols have a capability similar to Rust's
#[derive] proc-macros, but they are wired into the compiler and have to be RFC'd into the language on a case-by-case basis.
Finally, there's move semantics. Swift, like C++, copies value types (structs, enums, and primitive builtins) automatically, so there's no way to express single ownership reliably. Incidentally, value types containing references are also copied implicitly, so it's even easier to accidentally share ownership of a value.