The best solution to this problem seems to be Python's "with".
with open(oldfile, 'r', encoding='utf-8') as infile:
with open(newfile, 'w') as outfile:
....
The implied close will be executed on exiting the "with". "with" calls .__enter__ on the "open" object at entrance, and .__exit__ at exit. This works even if exit is via return or an exception.
If the implied close generates an exception, everything still works reasonably. __exit__ has a "traceback" parameter, which indicates an exception is in progress. An exit operation should do its closeout, then re-raise the exception sent as the traceback, if any. This works even when a destructor raises an exception. (A typical case is closing a network connection. If the other end dies, close will generate an error, which should not be lost.) The default use of "with" in Python thus comes with correct single and nested error handling.
That's not true of RAII or Go's "defer". In C++'s destructors, you can't return an error code, and an exception in a destructor is usually a big problem.
It's hard, but not impossible, for a deferred call in Go to report an error.
The deferred function can muck with the parameters of the function which called it. This only works
if the deferred function was written specifically for the context in which it was called; just deferring a "close" won't do this. So you don't get proper error handling by default.
(Languages seem to need exceptions. Both Go and Rust started with an always-aborts panic mechanism. That was too drastic in practice, and in both languages, panicking was made recoverable and provided with unwinding machinery. So both languages now have the heavy machinery for exceptions, without the language support for them. This results in kludges to simulate exceptions.)
> Languages seem to need exceptions. Both Go and Rust started with an always-aborts panic mechanism. That was too drastic in practice, and in both languages, panicking was made recoverable and provided with unwinding machinery. So both languages now have the heavy machinery for exceptions, without the language support for them. This results in kludges to simulate exceptions.
This is completely incorrect with respect to Rust, both historically and practically. Rust started with unwinding, and then grew opt-in abort machinery for those use cases where it matters. And, for what seems like the millionth time, unwinding does not equal exceptions; this is like saying that C has exceptions because of setjmp/longjmp. Unwinding exists in Rust solely to give one the ability to fail gracefully, running destructors that might have valuable side effects and allowing Rust-written subsystems to avoid taking down the entire system. For an example of the latter, Skylight, who ships a Ruby gem written in Rust, "catches" any unwinding so that an errant failure in their gem does not bring down a client's entire Ruby process. I have been writing Rust for years and, without exaggeration or (heh) exception, have never once seen the ability to "catch" unwinding used as a basis for constructing ad-hoc recoverable exception-handling, especially since abort-on-panic is an entirely valid mode for Rust applications (this is how Dropbox's backend Rust is compiled, for example) and completely obviates any sane attempt to use panics for recoverable error handling.
And the same goes for Go. The only legitimate use of panic is still always-abort; recover merely helps one define what part of the program has aborted, and is a noop except within a deferred call — you'd be hard pressed to use recover as casually as exceptions in Java, Python or Ruby. Typically, this is the case when a master/workers scenario is implemented[0], where the master has to survive a worker's death, because they're different "processes", but the latter definitely dies from panicking (possibly even caused by a call to a third party lib such as described by kibwen)
The situation is actually a bit more nuanced in Go; an idiomatic Go library may use panic/recover internally as a means of error handling, but is supposed to convert any panics to error codes for external consumption (IOW, panics should not be a part of any public API). You can see this pattern endorsed in this blog post: https://blog.golang.org/defer-panic-and-recover .
> The best solution to this problem seems to be Python's "with".
Ehhh... `with` is a crude substitute for destructors. C++ is a whole language designed around RAII. Python really is not.
You do make a good point that the handling of exceptions in destructors in C++ is problematic. But you can actually throw from destructors, as long as the destructor wasn't called during unwind for another throw. I'm of the opinion that the inability to throw again during unwind is a glaring bug in the standard, since it presumes people can know if their code is going to throw, which is like presuming people know where their bugs are. But the reasonable way to work around it is to squelch exceptions from destructors when another exception is already in-flight. This is usually fine, as the "secondary exceptions" thrown during an unwind are usually side-effects of the "primary exception" anyway.
Surely ruby's blocks are much better here. For one they're a much more general construct than with, and second exception handling is done with normal language constructs like rescue and ensure. Implementing one is simple and obvious and is nicely contained in a single method.
For example
File.open(args) do |file|
# use file
end
And the implementation is simply
def self.open(*args)
file = File.new(*args)
begin
yield file
ensure
file.close
end
end
I know Python far better than Ruby, but it looks like the semantics are equivalent, it's just that Ruby's standard `open` implementation includes the cleanup rather than leaving it up to the caller as in Python.
Ruby does this at the cost of having non-local returns from those blocks, which complicates semantics quite a bit (since now you need to distinguish closures that capture "return" and other control transfer from those that don't, and you need to handle the scenario where a control transfer happens when the target is already gone).
On the other hand, it means any helper/builder has to handle this pattern on its own rather than be able to return a file object which can be "context managed" by a caller, which Python allows.
I agree that Python's `with` is nicer than Go's `defer` (and I use both languages almost daily). I'm not sure I agree that languages seem to need exceptions--stack traces in the presence of errors are certainly handy, and there's certainly a case for panicking, and it's arguably more ergonomic to have syntax or type-system support for eliminating error-handling boilerplate, but I still prefer the simplicity of returned errors. That said, I'm really interested to hear different view points.
That's really complicated. The great thing about the "with" clause is that it Just Works. The easiest way to handle opens and closes is thus exception-safe.
What's wrong with indenting? You're entering a scope; indenting is appropriate.
I wrote "if that's an issue". For most cases it's not an issue.
The primary use case ExitStack is not to reduce indentation levels; it's to support "a variable number of context managers and other cleanup operations in a single with statement".
You wrote "The easiest way to handle opens and closes is thus exception-safe".
Consider if you need to merge up to 50 files containing sorted lines, where the result must also be sorted, and where you should not load all of the data into memory first. (Sorted according to UTF-8.) How would you do it? Not so easy, is it?
With ExitStack you could write it as something like the following sketch:
filenames = [ list, of, file, names]
with ExitStack() as stack:
files = [stack.enter_context(open(fname, "rb")) for fname in filenames]
with open("merged_lines.txt", "wb") as output:
output.writelines(heapq.merge(*files))
As you say, the great thing about this with clause is that it Just Works.
A side-effect of making ExitStack work is that it might help if the indentation depth is a problem, which mappu apparently has experienced.
The variable isn't necessary, and contextlib.contextmanager is the standard wrapper for arbitrary functions. Here's the example code from the documentation:
from contextlib import contextmanager
@contextmanager
def tag(name):
print "<%s>" % name
yield
print "</%s>" % name
>>> with tag("h1"):
... print("foo")
...
<h1>
foo
</h1>
But how does that close over other locals? I'll admit I haven't looked too closely, but go defer makes it pretty easy to drop in a lambda. Getting stuff into your decorated functions by reference is going to be awkward, unless I'm missing something.
I'll need to see an example to understand the issue. The idioms between the two languages are different enough that idiomatic Go might be written a different way for idiomatic Python.
Remember, "with" isn't always required - you can also write the code using a try/finally.
If the implied close generates an exception, everything still works reasonably. __exit__ has a "traceback" parameter, which indicates an exception is in progress. An exit operation should do its closeout, then re-raise the exception sent as the traceback, if any. This works even when a destructor raises an exception. (A typical case is closing a network connection. If the other end dies, close will generate an error, which should not be lost.) The default use of "with" in Python thus comes with correct single and nested error handling.
That's not true of RAII or Go's "defer". In C++'s destructors, you can't return an error code, and an exception in a destructor is usually a big problem.
It's hard, but not impossible, for a deferred call in Go to report an error. The deferred function can muck with the parameters of the function which called it. This only works if the deferred function was written specifically for the context in which it was called; just deferring a "close" won't do this. So you don't get proper error handling by default.
(Languages seem to need exceptions. Both Go and Rust started with an always-aborts panic mechanism. That was too drastic in practice, and in both languages, panicking was made recoverable and provided with unwinding machinery. So both languages now have the heavy machinery for exceptions, without the language support for them. This results in kludges to simulate exceptions.)